Oct 30 21:25:36.476505 ip-10-0-137-168 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:25:36.476516 ip-10-0-137-168 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:25:36.476524 ip-10-0-137-168 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:25:36.476737 ip-10-0-137-168 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:25:46.541808 ip-10-0-137-168 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:25:46.541823 ip-10-0-137-168 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 419bfb6a37b04924914912f297e0be26 -- Oct 30 21:27:47.121215 ip-10-0-137-168 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:27:47.528410 ip-10-0-137-168 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:27:47.528410 ip-10-0-137-168 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:27:47.528410 ip-10-0-137-168 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:27:47.528410 ip-10-0-137-168 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:27:47.533497 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533423 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.533497 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533498 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533502 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533506 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533508 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533511 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533514 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533517 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533519 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533524 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533529 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533533 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533536 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533538 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533541 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533544 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533546 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533549 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533551 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533554 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.533562 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533557 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533559 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533562 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533564 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533567 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533570 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533573 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533585 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533589 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533591 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533594 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533596 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533599 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533601 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533604 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533606 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533609 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533611 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533614 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533616 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.534021 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533619 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533621 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533624 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533627 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533632 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533635 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533638 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533641 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533644 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533646 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533649 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533651 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533654 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533657 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533660 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533663 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533665 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533668 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533670 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.534519 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533673 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533676 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533679 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533681 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533684 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533686 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533689 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533691 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533694 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533696 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533698 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533701 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533704 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533706 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533709 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533711 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533713 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533716 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533719 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533721 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.534966 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533724 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533727 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533730 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533732 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533735 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533737 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533740 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533743 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533745 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533748 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533750 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533753 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533755 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533758 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533761 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533764 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533766 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533769 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533771 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.535435 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533774 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533777 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533779 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533782 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533784 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533787 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533790 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.533792 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534532 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534539 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534542 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534545 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534548 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534550 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534554 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534556 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534559 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534561 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534564 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534567 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.535902 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534571 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534575 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534579 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534581 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534584 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534587 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534589 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534592 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534595 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534598 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534601 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534604 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534606 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534609 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534612 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534614 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534617 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534619 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534622 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.536390 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534625 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534627 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534630 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534633 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534636 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534638 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534641 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534644 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534647 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534649 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534652 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534654 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534657 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534660 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534663 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534665 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534668 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534670 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534673 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534676 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.536832 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534678 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534681 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534684 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534687 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534689 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534692 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534694 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534696 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534699 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534702 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534704 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534706 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534709 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534711 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534714 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534717 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534720 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534722 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534725 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.537353 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534727 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534730 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534732 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534735 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534738 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534741 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534743 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534746 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534749 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534752 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534754 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534757 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534759 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534761 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534764 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534766 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534769 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534772 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534774 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534777 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.537797 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534779 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534782 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534784 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534787 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534789 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534792 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534794 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534797 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534800 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534802 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534805 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534807 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534810 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534812 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534815 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.534819 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535314 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535321 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535324 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.538289 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535327 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535330 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535333 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535335 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535338 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535340 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535343 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535346 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535348 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535353 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535357 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535360 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535363 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535366 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535368 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535371 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535373 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535376 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535378 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.538722 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535381 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535384 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535395 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535398 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535401 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535403 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535406 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535409 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535412 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535414 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535417 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535419 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535422 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535424 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535428 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535432 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535436 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535439 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535442 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535445 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.539177 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535447 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535450 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535452 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535455 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535458 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535461 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535464 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535466 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535469 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535471 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535474 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535476 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535479 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535481 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535484 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535487 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535490 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535492 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535494 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535498 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.539645 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535501 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535503 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535506 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535508 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535511 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535513 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535516 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535518 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535520 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535523 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535525 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535528 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535530 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535533 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535535 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535538 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535541 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535545 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535548 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.540128 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535550 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535556 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535558 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535561 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535563 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535566 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535569 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535572 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535574 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535577 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535579 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535582 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535584 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535587 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535590 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535592 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535594 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535597 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535600 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.540570 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535602 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535604 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535608 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535610 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.535613 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536571 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536581 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536586 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536591 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536595 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536601 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536605 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536609 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536613 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536616 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536620 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536623 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536628 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536631 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536634 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536637 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536640 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536643 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:27:47.541055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536647 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536650 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536653 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536656 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536660 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536664 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536667 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536670 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536673 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536676 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536679 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536682 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536685 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536687 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536692 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536695 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536698 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536700 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536703 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536708 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536712 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536716 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536719 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536722 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:27:47.541665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536725 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536729 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536733 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536737 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536740 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536743 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536746 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536749 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536752 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536754 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536757 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536760 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536764 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536767 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536770 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536774 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536777 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536780 2580 flags.go:64] FLAG: --help="false" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536783 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536786 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536789 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536792 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536795 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:27:47.542317 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536798 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536801 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536804 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536807 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536811 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536814 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536818 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536821 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536824 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536827 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536830 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536835 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536838 2580 flags.go:64] FLAG: --lock-file="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536841 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536844 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536847 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536853 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536855 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536858 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536861 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536864 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536868 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536871 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536873 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536883 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:27:47.542893 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536886 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536897 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536900 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536903 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536906 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536909 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536911 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536914 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536917 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536925 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536928 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536932 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536935 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536938 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536941 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536957 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536962 2580 flags.go:64] FLAG: --port="10250" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536967 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536974 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ec740e4f156883e0" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536977 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536980 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536983 2580 flags.go:64] FLAG: --register-node="true" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536986 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536990 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536993 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.536996 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:27:47.543494 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537000 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537003 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537007 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537009 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537012 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537015 2580 flags.go:64] FLAG: --runonce="false" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537018 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537021 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537024 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537027 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537030 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537033 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537035 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537038 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537041 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537044 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537047 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537050 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537053 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537056 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537059 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537064 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537067 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537071 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537076 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:27:47.544117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537080 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537083 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537086 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537089 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537092 2580 flags.go:64] FLAG: --v="2" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537096 2580 flags.go:64] FLAG: --version="false" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537100 2580 flags.go:64] FLAG: --vmodule="" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537104 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537108 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537271 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537276 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537279 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537282 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537285 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537288 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537290 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537293 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537296 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537298 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537301 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537303 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537306 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.544693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537308 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537311 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537314 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537317 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537319 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537322 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537325 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537329 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537333 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537337 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537339 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537342 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537345 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537348 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537350 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537353 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537355 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537358 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537361 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537364 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.545216 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537367 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537369 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537372 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537375 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537377 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537380 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537383 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537385 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537388 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537390 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537393 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537395 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537398 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537400 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537403 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537405 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537408 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537411 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537413 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537416 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.545681 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537419 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537422 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537425 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537427 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537430 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537432 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537435 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537437 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537440 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537442 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537445 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537447 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537450 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537452 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537455 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537457 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537463 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537466 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537470 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.546240 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537473 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537476 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537479 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537482 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537484 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537487 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537489 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537491 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537494 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537496 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537499 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537502 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537505 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537507 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537512 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537514 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537517 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537520 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537522 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.546693 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537525 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537527 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537530 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537533 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537535 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537538 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537541 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537543 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537546 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537549 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537552 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537555 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537557 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537560 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537563 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.537589 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:27:47.547145 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537623 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537628 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537631 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537634 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537636 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537639 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537642 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537644 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537647 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537649 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537652 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537654 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537659 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537662 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537665 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537667 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537670 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537672 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537675 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537677 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.547520 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537680 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537682 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537685 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537688 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537690 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537693 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537697 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537699 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537702 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537704 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537707 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537709 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537712 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537714 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537719 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537723 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537726 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537729 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537732 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.547985 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537735 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537737 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537740 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537743 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537745 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537749 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537752 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537755 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537757 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537760 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537763 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537765 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537768 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537770 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537773 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537775 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537778 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537780 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537783 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.548429 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537787 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537789 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537792 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537794 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537797 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537799 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537802 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537805 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537808 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537810 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537813 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537815 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537818 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537821 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537823 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537826 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537829 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537832 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537835 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537837 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.548893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537840 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537843 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537845 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537848 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537850 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537853 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537855 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537858 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537860 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537862 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537865 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537867 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537871 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537875 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537879 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537882 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537884 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537887 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537890 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.549351 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537892 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537895 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537897 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537900 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537905 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537907 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537911 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.537913 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.544877 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.544892 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.544909 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.544914 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545003 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545010 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545017 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545020 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545023 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.549778 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545025 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545028 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545037 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545040 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545043 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545045 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545048 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545051 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545054 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545056 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545059 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545061 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545064 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545066 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545069 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545072 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545074 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545077 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545079 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545082 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.550246 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545085 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545087 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545090 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545094 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545096 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545099 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545101 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545104 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545106 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545109 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545112 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545114 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545117 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545119 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545122 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545130 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545139 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545142 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545144 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545147 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.550716 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545149 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545152 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545154 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545156 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545159 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545162 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545164 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545166 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545169 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545171 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545174 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545176 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545178 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545181 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545184 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545187 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545190 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545193 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545195 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.551264 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545198 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545201 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545203 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545206 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545208 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545211 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545215 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545219 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545222 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545230 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545234 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545237 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545240 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545242 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545245 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545247 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545250 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545252 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545255 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.551705 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545257 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545260 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545262 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545265 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545268 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545271 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545273 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545276 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545279 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545282 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545285 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545287 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545289 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545292 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545294 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545297 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545299 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545302 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545304 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545307 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.552343 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545309 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545312 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.545341 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545569 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545576 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545581 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545584 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545587 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545590 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545593 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545595 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545598 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545600 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545603 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545606 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.553084 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545608 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545611 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545613 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545616 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545618 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545621 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545623 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545626 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545629 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545631 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545634 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545636 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545639 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545641 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545644 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545647 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545649 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545652 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545654 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545657 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.553465 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545659 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545672 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545675 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545677 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545680 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545682 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545685 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545688 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545691 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545694 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545696 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545699 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545701 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545704 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545707 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545710 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545712 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545715 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545717 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.554212 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545720 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545723 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545725 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545727 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545730 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545733 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545736 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545739 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545742 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545744 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545747 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545749 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545751 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545754 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545756 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545764 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545767 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545770 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545772 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.554893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545774 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545777 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545779 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545783 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545785 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545788 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545790 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545793 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545796 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545798 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545801 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545803 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545806 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545808 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545811 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545813 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545815 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545818 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545820 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545823 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.555433 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545825 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545828 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545830 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545832 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545835 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545837 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545840 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545842 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545845 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545853 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545856 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545858 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545861 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545863 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545865 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545868 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.556005 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.545893 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545939 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545966 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545970 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545973 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545976 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545979 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545981 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545984 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545986 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545989 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545991 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545994 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545996 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.545999 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546002 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546004 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546007 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546009 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546012 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:47.556378 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546014 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546017 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546019 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546022 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546024 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546027 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546039 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546042 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546044 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546047 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546050 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546053 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546055 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546058 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546060 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546065 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546068 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546071 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546074 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:47.557095 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546076 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546079 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546081 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546083 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546086 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546088 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546091 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546093 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546096 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546098 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546101 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546103 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546105 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546108 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546110 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546118 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546121 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546124 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546126 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546129 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:47.557539 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546136 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546139 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546142 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546145 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546147 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546150 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546152 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546155 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546157 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546160 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546162 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546164 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546167 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546169 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546172 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546174 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546177 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546179 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546182 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:47.558044 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546184 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546187 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546192 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546195 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546198 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546200 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546203 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546205 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546209 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546214 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546217 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546220 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546222 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546225 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546233 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546235 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546238 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546241 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546243 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:47.558481 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546246 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546249 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546252 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546254 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546257 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546259 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546262 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546264 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546266 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:47.546269 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.546327 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.548370 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.549747 2580 server.go:1014] "Starting client certificate rotation" Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.550244 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.550279 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.553128 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:27:47.558988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.556939 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:27:47.566588 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.566573 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:27:47.569750 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.569733 2580 log.go:25] "Validated CRI v1 image API" Oct 30 21:27:47.570588 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.570560 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:27:47.575976 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.575939 2580 fs.go:135] Filesystem UUIDs: map[2efac825-e414-46f9-a17e-a23aac127836:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b2b6838c-4110-468b-80c0-c491f8f690a9:/dev/nvme0n1p3] Oct 30 21:27:47.576052 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.575976 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:27:47.581340 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.581234 2580 manager.go:217] Machine: {Timestamp:2026-10-30 21:27:47.579868255 +0000 UTC m=+0.341194876 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111880 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22783dd70c706f19907f7e05892a9e SystemUUID:ec22783d-d70c-706f-1990-7f7e05892a9e BootID:419bfb6a-37b0-4924-9149-12f297e0be26 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:c0:9e:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:c0:9e:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:09:33:8d:f0:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:27:47.581427 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.581343 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:27:47.581465 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.581452 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:27:47.585049 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.585027 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:27:47.585191 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.585052 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-168.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:27:47.585234 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.585197 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:27:47.585234 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.585209 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:27:47.585289 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.585248 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:27:47.585289 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.585266 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:27:47.586315 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.586305 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:27:47.589621 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.589602 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:27:47.590359 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590347 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:27:47.590484 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590475 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:27:47.590512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590508 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:27:47.590541 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590524 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:27:47.590541 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590535 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:27:47.590593 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590544 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:27:47.590593 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590554 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:27:47.590861 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.590849 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:27:47.593646 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.593627 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:27:47.599845 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.599828 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:27:47.600552 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600540 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:27:47.600595 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600570 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:27:47.600717 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600708 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:27:47.600749 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600723 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:27:47.600749 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600730 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:27:47.600835 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600766 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:27:47.600835 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600773 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:27:47.600835 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.600778 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:27:47.601441 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601432 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:27:47.601441 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601440 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:27:47.601504 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601448 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:27:47.601504 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601453 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:27:47.601504 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601462 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:27:47.601504 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601471 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:27:47.601919 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601910 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:27:47.601962 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.601924 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:27:47.604367 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.604352 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-168.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:27:47.605858 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.605845 2580 server.go:1263] "Started kubelet" Oct 30 21:27:47.605961 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.605904 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:27:47.606017 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.605937 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:27:47.606073 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.606031 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:27:47.606638 ip-10-0-137-168 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:27:47.607111 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.607095 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:27:47.608031 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.608014 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:27:47.610298 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.610279 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:27:47.611807 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.611783 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:27:47.612406 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.612390 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-168.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:27:47.612759 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.611448 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-168.us-east-2.compute.internal.18e36bdb9d5e7d20 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-168.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-168.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:27:47.60581456 +0000 UTC m=+0.367141180,LastTimestamp:2026-10-30 21:27:47.60581456 +0000 UTC m=+0.367141180,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-168.us-east-2.compute.internal,}" Oct 30 21:27:47.613290 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.613265 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvfjj" Oct 30 21:27:47.613861 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.613845 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:27:47.614772 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.614756 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:27:47.614893 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.614872 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:47.615330 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615310 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:27:47.615330 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615312 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:27:47.615439 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615351 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:27:47.615483 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615447 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:27:47.615483 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615456 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:27:47.615721 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615706 2580 factory.go:55] Registering systemd factory Oct 30 21:27:47.615721 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615723 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:27:47.615983 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615969 2580 factory.go:153] Registering CRI-O factory Oct 30 21:27:47.616030 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.615986 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 21:27:47.616067 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.616055 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:27:47.616100 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.616070 2580 factory.go:103] Registering Raw factory Oct 30 21:27:47.616100 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.616081 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 21:27:47.617119 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.617105 2580 manager.go:319] Starting recovery of all containers Oct 30 21:27:47.620267 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.620240 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-168.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:27:47.621544 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.621523 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvfjj" Oct 30 21:27:47.628595 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.628566 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:27:47.631682 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.631661 2580 manager.go:324] Recovery completed Oct 30 21:27:47.635679 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.635662 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.638093 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638075 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.638155 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638113 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.638155 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638129 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:47.638737 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638719 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:27:47.638737 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638736 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:27:47.638876 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638756 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:27:47.638876 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.638725 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.640237 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.640219 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.640324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.640245 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.640324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.640256 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:47.640906 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.640890 2580 policy_none.go:50] "Start" Oct 30 21:27:47.640989 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.640909 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:27:47.640989 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.640919 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:27:47.643199 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.643188 2580 policy_none.go:44] "Start" Oct 30 21:27:47.677509 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.677465 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:27:47.682060 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682048 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 21:27:47.682111 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.682073 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:27:47.682111 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682082 2580 server.go:87] "Starting device plugin registration server" Oct 30 21:27:47.682280 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682269 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:27:47.682335 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682279 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:27:47.682442 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682420 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:27:47.682592 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682512 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:27:47.682592 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.682525 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:27:47.683140 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.683122 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:27:47.683213 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.683177 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:47.782406 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.782327 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.784328 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.783162 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.784328 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.783187 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.784328 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.783199 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:47.784328 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.783224 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:47.790188 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.790174 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:47.790233 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.790195 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-168.us-east-2.compute.internal\": node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:47.809106 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.809085 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:47.888869 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.888849 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:27:47.890004 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.889989 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:27:47.890072 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.890009 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:27:47.890072 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.890033 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:27:47.890999 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.890061 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:27:47.896991 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.896975 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:47.910049 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:47.910036 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:47.991529 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.991500 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal"] Oct 30 21:27:47.991613 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.991563 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.992327 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.992313 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.992385 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.992341 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.992385 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.992352 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:47.994456 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.994445 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.994615 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.994602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:47.994653 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.994630 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.995257 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.995238 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.995324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.995260 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.995324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.995272 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:47.995324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.995242 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.995324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.995305 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.995324 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.995319 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:47.997238 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.997221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:47.997339 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.997245 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:47.997928 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.997914 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:47.998010 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.997938 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:47.998010 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:47.997960 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:48.010425 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.010408 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:48.018249 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.018231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.018322 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.018254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.018322 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.018271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de6e1dbd7017e381fdeb008c14987499-config\") pod \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"de6e1dbd7017e381fdeb008c14987499\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.027190 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.027175 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.029811 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.029794 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.110904 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.110865 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:48.119222 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.119208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.119269 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.119232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.119269 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.119248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de6e1dbd7017e381fdeb008c14987499-config\") pod \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"de6e1dbd7017e381fdeb008c14987499\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.119367 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.119294 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.119367 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.119310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de6e1dbd7017e381fdeb008c14987499-config\") pod \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"de6e1dbd7017e381fdeb008c14987499\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.119367 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.119349 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.211374 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.211355 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:48.311849 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.311823 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:48.330302 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.330287 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.333231 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.333215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.412404 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.412249 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:48.512704 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.512683 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:27:48.549101 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.549086 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:27:48.549431 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.549206 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:27:48.581832 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.581816 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:48.591104 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.591088 2580 apiserver.go:52] "Watching apiserver" Oct 30 21:27:48.600793 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.600767 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:27:48.601110 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.601092 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-sq9rx","openshift-network-operator/iptables-alerter-bc4bx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm","openshift-cluster-node-tuning-operator/tuned-vnxqg","openshift-multus/network-metrics-daemon-p2nkx","kube-system/global-pull-secret-syncer-mq6jf","kube-system/konnectivity-agent-mcglb","openshift-ovn-kubernetes/ovnkube-node-wnjz7","openshift-dns/node-resolver-r7866","openshift-multus/multus-additional-cni-plugins-87ql4","openshift-image-registry/node-ca-8l22v","openshift-multus/multus-nx794"] Oct 30 21:27:48.603533 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.603518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:48.603600 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.603582 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:27:48.605735 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.605716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.608345 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.608327 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.610125 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.610108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.612360 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.612347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:48.612418 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.612401 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:27:48.613440 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.613415 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:27:48.613537 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.613499 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.613850 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.613834 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.613928 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.613904 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:27:48.616049 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.614759 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:27:48.616049 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.614862 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.616049 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.614985 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.616049 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.615120 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxqgz\"" Oct 30 21:27:48.617135 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.617110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.617242 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.617138 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g2c5m\"" Oct 30 21:27:48.617648 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.617617 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:27:48.619877 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.619856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.621176 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-tuned\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621176 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.621176 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.621342 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysconfig\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621342 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:48.621342 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b6c776-a4e5-492f-a017-3c9dfc567d56-host-slash\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.621342 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.621473 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-modprobe-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621473 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621473 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5k6\" (UniqueName: \"kubernetes.io/projected/75908611-03b2-42df-85e5-128bd14c5a43-kube-api-access-ng5k6\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:48.621577 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.621577 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-device-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.621657 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-var-lib-kubelet\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621657 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v254t\" (UniqueName: \"kubernetes.io/projected/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-kube-api-access-v254t\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621657 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-lib-modules\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621793 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-tmp\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621793 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b6c776-a4e5-492f-a017-3c9dfc567d56-iptables-alerter-script\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.621793 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.621908 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-run\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.621908 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-kubelet-config\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.621908 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-dbus\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.622065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wgs\" (UniqueName: \"kubernetes.io/projected/55b6c776-a4e5-492f-a017-3c9dfc567d56-kube-api-access-x5wgs\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.622065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.622065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.621995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxshq\" (UniqueName: \"kubernetes.io/projected/5dd38803-1d13-470a-97b1-99bc9262d8f0-kube-api-access-jxshq\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.622065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-kubernetes\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.622194 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:48.622194 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-conf\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.622298 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-host\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.622298 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-systemd\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.622298 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-sys\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.622419 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.622325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.624614 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.624596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.626967 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.626918 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:22:48 +0000 UTC" deadline="2028-07-03 09:11:28.569150843 +0000 UTC" Oct 30 21:27:48.626967 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.626964 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14675h43m39.942189645s" Oct 30 21:27:48.627141 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.626973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.629319 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.629304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.631653 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.631638 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nx794" Oct 30 21:27:48.632775 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.632760 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal"] Oct 30 21:27:48.640292 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.640275 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:27:48.642539 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.642524 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:27:48.642625 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.642603 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:27:48.654237 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.654220 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:27:48.654385 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.654370 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal"] Oct 30 21:27:48.701447 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.701429 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgspz" Oct 30 21:27:48.707959 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.707942 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgspz" Oct 30 21:27:48.716237 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.716220 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:27:48.722045 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722026 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.722144 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722089 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.722275 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722260 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:27:48.722380 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722362 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:27:48.722440 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e6b899c-d914-4adb-9a7c-230e359b1497-agent-certs\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.722753 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722729 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99k6q\"" Oct 30 21:27:48.722826 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-lib-modules\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.722826 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-tmp\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.722933 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.722933 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-lib-modules\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.722933 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722901 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.723097 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.722955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-run\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.723097 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.723097 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-k8s-cni-cncf-io\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.723097 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.723097 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723089 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.723336 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wgs\" (UniqueName: \"kubernetes.io/projected/55b6c776-a4e5-492f-a017-3c9dfc567d56-kube-api-access-x5wgs\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.723336 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723127 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-run\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.723336 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723217 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.723336 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.723521 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-host\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.723521 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-multus\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.723521 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-kubelet\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.723521 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-systemd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.723711 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.723711 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-conf\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.723711 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-host\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.723711 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-bin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.723903 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclrn\" (UniqueName: \"kubernetes.io/projected/4422cee8-aec1-4569-9277-bba73b00a9ad-kube-api-access-hclrn\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.723903 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b899c-d914-4adb-9a7c-230e359b1497-konnectivity-ca\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.723903 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723791 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:27:48.724062 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723932 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:27:48.724062 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.723995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-systemd\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.724062 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-system-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.724204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-cnibin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.724204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724095 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:27:48.724204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-daemon-config\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.724204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-etc-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.724204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-socket-dir-parent\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.724204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724208 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pcx6\"" Oct 30 21:27:48.724497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724308 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-host\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.724497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724391 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.724602 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724499 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:27:48.724657 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724558 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-conf\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.724713 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724673 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.724760 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724709 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d996q\"" Oct 30 21:27:48.724836 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724818 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:27:48.724888 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724835 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:27:48.724939 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-sys\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.724998 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.724964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-tuned\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.725070 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-systemd\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.725130 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725085 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.725130 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.725236 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725142 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:27:48.725236 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725216 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pqqf8\"" Oct 30 21:27:48.725339 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725268 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.725339 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725302 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-sys\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.725447 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.725503 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725473 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.725503 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725493 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.725612 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysconfig\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.725670 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-serviceca\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.725670 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysconfig\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.725770 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-script-lib\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.725770 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.725917 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-hostroot\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.725977 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.726057 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.725978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.726116 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.726207 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cnibin\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.726263 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-device-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.726347 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726276 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v254t\" (UniqueName: \"kubernetes.io/projected/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-kube-api-access-v254t\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-netd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovn-node-metrics-cert\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9233cc4d-86f9-4890-9464-33c7eefd42a8-tmp-dir\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b6c776-a4e5-492f-a017-3c9dfc567d56-iptables-alerter-script\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726736 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726815 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-kubelet-config\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-dbus\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-conf-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.726972 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727011 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727171 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-flwfj\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-node-log\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727288 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8x7v\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727322 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-system-cni-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727366 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxshq\" (UniqueName: \"kubernetes.io/projected/5dd38803-1d13-470a-97b1-99bc9262d8f0-kube-api-access-jxshq\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.727815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727537 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727538 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-kubernetes\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-kubernetes\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727729 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-device-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.727917 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b6c776-a4e5-492f-a017-3c9dfc567d56-iptables-alerter-script\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728057 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728576 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ct8zc\"" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds64h\" (UniqueName: \"kubernetes.io/projected/9233cc4d-86f9-4890-9464-33c7eefd42a8-kube-api-access-ds64h\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-netns\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728736 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-kubelet-config\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-etc-kubernetes\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-kubelet\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.728927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-systemd-units\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktlq\" (UniqueName: \"kubernetes.io/projected/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-kube-api-access-lktlq\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddgd\" (UniqueName: \"kubernetes.io/projected/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-kube-api-access-6ddgd\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.728974 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-dbus\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-env-overrides\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-multus-certs\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-log-socket\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-config\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.729232 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.729307 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:49.229287758 +0000 UTC m=+1.990614387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b6c776-a4e5-492f-a017-3c9dfc567d56-host-slash\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-netns\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-slash\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b6c776-a4e5-492f-a017-3c9dfc567d56-host-slash\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.729547 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqwg\" (UniqueName: \"kubernetes.io/projected/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-kube-api-access-sbqwg\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9233cc4d-86f9-4890-9464-33c7eefd42a8-hosts-file\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-modprobe-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5k6\" (UniqueName: \"kubernetes.io/projected/75908611-03b2-42df-85e5-128bd14c5a43-kube-api-access-ng5k6\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729581 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-modprobe-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729646 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-tuned\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729711 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-tmp\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-var-lib-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.729790 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-ovn\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.729851 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:49.229839131 +0000 UTC m=+1.991165740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-var-lib-kubelet\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.729963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-cni-binary-copy\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.730009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-bin\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.730075 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.730009 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-var-lib-kubelet\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.736349 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.736329 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wgs\" (UniqueName: \"kubernetes.io/projected/55b6c776-a4e5-492f-a017-3c9dfc567d56-kube-api-access-x5wgs\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.745017 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.744992 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:48.745017 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.745015 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:48.745017 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.745028 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:48.745242 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:48.745115 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:49.24509195 +0000 UTC m=+2.006418573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:48.745467 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.745448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v254t\" (UniqueName: \"kubernetes.io/projected/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-kube-api-access-v254t\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.747528 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.747507 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5k6\" (UniqueName: \"kubernetes.io/projected/75908611-03b2-42df-85e5-128bd14c5a43-kube-api-access-ng5k6\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:48.747609 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.747552 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxshq\" (UniqueName: \"kubernetes.io/projected/5dd38803-1d13-470a-97b1-99bc9262d8f0-kube-api-access-jxshq\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.830696 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-bin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.830823 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclrn\" (UniqueName: \"kubernetes.io/projected/4422cee8-aec1-4569-9277-bba73b00a9ad-kube-api-access-hclrn\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.830823 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b899c-d914-4adb-9a7c-230e359b1497-konnectivity-ca\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.830823 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830771 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-bin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.830823 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-system-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-cnibin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-daemon-config\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-etc-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-socket-dir-parent\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-serviceca\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-script-lib\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-hostroot\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830971 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-socket-dir-parent\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830995 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-cnibin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.830986 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-etc-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831025 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-hostroot\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831058 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-system-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cnibin\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-netd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovn-node-metrics-cert\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9233cc4d-86f9-4890-9464-33c7eefd42a8-tmp-dir\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-conf-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-node-log\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-system-cni-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds64h\" (UniqueName: \"kubernetes.io/projected/9233cc4d-86f9-4890-9464-33c7eefd42a8-kube-api-access-ds64h\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-netns\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-etc-kubernetes\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.831444 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831440 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-node-log\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831488 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-system-cni-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831527 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-kubelet\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831546 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-conf-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831580 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-netns\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831608 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-etc-kubernetes\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831630 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cnibin\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831656 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-kubelet\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-systemd-units\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831730 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-systemd-units\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-netd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktlq\" (UniqueName: \"kubernetes.io/projected/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-kube-api-access-lktlq\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddgd\" (UniqueName: \"kubernetes.io/projected/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-kube-api-access-6ddgd\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-env-overrides\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-multus-certs\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.831859 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-multus-certs\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-log-socket\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832049 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-config\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832165 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-daemon-config\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-log-socket\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-netns\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-slash\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqwg\" (UniqueName: \"kubernetes.io/projected/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-kube-api-access-sbqwg\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9233cc4d-86f9-4890-9464-33c7eefd42a8-hosts-file\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832361 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-slash\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-var-lib-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-ovn\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832402 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-netns\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-var-lib-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832453 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9233cc4d-86f9-4890-9464-33c7eefd42a8-hosts-file\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832487 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-ovn\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.832771 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-cni-binary-copy\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832563 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-bin\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e6b899c-d914-4adb-9a7c-230e359b1497-agent-certs\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-k8s-cni-cncf-io\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832688 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-script-lib\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-host\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-multus\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9233cc4d-86f9-4890-9464-33c7eefd42a8-tmp-dir\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-kubelet\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-systemd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832850 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b899c-d914-4adb-9a7c-230e359b1497-konnectivity-ca\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832865 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-bin\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.833663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832952 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-k8s-cni-cncf-io\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-systemd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.832957 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-kubelet\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-multus\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833127 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-host\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833154 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833173 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833295 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-cni-binary-copy\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833440 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-serviceca\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833547 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-config\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.834172 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.833765 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-env-overrides\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.835015 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.834994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovn-node-metrics-cert\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.835331 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.835314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e6b899c-d914-4adb-9a7c-230e359b1497-agent-certs\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:48.844307 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.844276 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds64h\" (UniqueName: \"kubernetes.io/projected/9233cc4d-86f9-4890-9464-33c7eefd42a8-kube-api-access-ds64h\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:48.844425 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.844395 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclrn\" (UniqueName: \"kubernetes.io/projected/4422cee8-aec1-4569-9277-bba73b00a9ad-kube-api-access-hclrn\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:27:48.844718 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.844697 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktlq\" (UniqueName: \"kubernetes.io/projected/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-kube-api-access-lktlq\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:48.845017 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.844994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddgd\" (UniqueName: \"kubernetes.io/projected/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-kube-api-access-6ddgd\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:48.845352 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.845334 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqwg\" (UniqueName: \"kubernetes.io/projected/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-kube-api-access-sbqwg\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:48.922755 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.922679 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:48.939055 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.939026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:27:48.960058 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.960032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:27:48.970088 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.970071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:27:48.986597 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:48.986577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:49.003768 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.003745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:27:49.011827 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.011805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r7866" Oct 30 21:27:49.014893 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.014858 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6670d9ca0a34b37e015f8cc739976c92.slice/crio-f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e WatchSource:0}: Error finding container f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e: Status 404 returned error can't find the container with id f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e Oct 30 21:27:49.015250 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.015223 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b899c_d914_4adb_9a7c_230e359b1497.slice/crio-f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e WatchSource:0}: Error finding container f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e: Status 404 returned error can't find the container with id f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e Oct 30 21:27:49.016202 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.016178 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f25b78_f285_4deb_a96f_b94c4bc6e9f4.slice/crio-61f6ee419cb3af5e21685fc2260dc4c7e08e1dc88e885ef1b37fa1143f26108d WatchSource:0}: Error finding container 61f6ee419cb3af5e21685fc2260dc4c7e08e1dc88e885ef1b37fa1143f26108d: Status 404 returned error can't find the container with id 61f6ee419cb3af5e21685fc2260dc4c7e08e1dc88e885ef1b37fa1143f26108d Oct 30 21:27:49.017924 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.017901 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6e1dbd7017e381fdeb008c14987499.slice/crio-c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6 WatchSource:0}: Error finding container c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6: Status 404 returned error can't find the container with id c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6 Oct 30 21:27:49.021921 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.021898 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:27:49.027590 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.027567 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9233cc4d_86f9_4890_9464_33c7eefd42a8.slice/crio-cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54 WatchSource:0}: Error finding container cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54: Status 404 returned error can't find the container with id cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54 Oct 30 21:27:49.046132 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.046112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:27:49.047635 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.047617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:27:49.049890 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.049870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nx794" Oct 30 21:27:49.056373 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.056348 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89806e93_18c5_43a5_b1ed_0f6eae4f20f3.slice/crio-0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa WatchSource:0}: Error finding container 0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa: Status 404 returned error can't find the container with id 0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa Oct 30 21:27:49.060255 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.060233 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ae92ae_3d8d_4ea7_bb9f_88d481fc0ada.slice/crio-4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658 WatchSource:0}: Error finding container 4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658: Status 404 returned error can't find the container with id 4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658 Oct 30 21:27:49.062061 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.062036 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4422cee8_aec1_4569_9277_bba73b00a9ad.slice/crio-7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041 WatchSource:0}: Error finding container 7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041: Status 404 returned error can't find the container with id 7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041 Oct 30 21:27:49.152485 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.152452 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:49.236667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.236589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:49.236667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.236642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:49.236667 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.236643 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:49.236859 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.236702 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:50.236687952 +0000 UTC m=+2.998014583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:49.236859 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.236759 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:49.236859 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.236802 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:50.236791239 +0000 UTC m=+2.998117847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:49.337272 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.337244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:49.337400 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.337360 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:49.337400 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.337376 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:49.337400 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.337385 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:49.337496 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.337427 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:50.33741601 +0000 UTC m=+3.098742619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:49.348211 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.348187 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b6c776_a4e5_492f_a017_3c9dfc567d56.slice/crio-3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644 WatchSource:0}: Error finding container 3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644: Status 404 returned error can't find the container with id 3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644 Oct 30 21:27:49.419221 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:27:49.419197 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd38803_1d13_470a_97b1_99bc9262d8f0.slice/crio-e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07 WatchSource:0}: Error finding container e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07: Status 404 returned error can't find the container with id e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07 Oct 30 21:27:49.422153 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.422124 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:49.711050 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.710911 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:22:49 +0000 UTC" deadline="2028-06-16 06:34:45.10884218 +0000 UTC" Oct 30 21:27:49.711050 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.710993 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14265h6m55.39785419s" Oct 30 21:27:49.895841 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.895240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:49.895841 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:49.895365 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:27:49.915658 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.915601 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" event={"ID":"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4","Type":"ContainerStarted","Data":"61f6ee419cb3af5e21685fc2260dc4c7e08e1dc88e885ef1b37fa1143f26108d"} Oct 30 21:27:49.929102 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.928984 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" event={"ID":"5dd38803-1d13-470a-97b1-99bc9262d8f0","Type":"ContainerStarted","Data":"e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07"} Oct 30 21:27:49.968099 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.967992 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nx794" event={"ID":"4422cee8-aec1-4569-9277-bba73b00a9ad","Type":"ContainerStarted","Data":"7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041"} Oct 30 21:27:49.983725 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:49.983696 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8l22v" event={"ID":"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada","Type":"ContainerStarted","Data":"4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658"} Oct 30 21:27:50.003454 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.003423 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"e7761f3f0956a26e38bf54f450c1cb28638e0633a2a158ff7f3892f146942b13"} Oct 30 21:27:50.023275 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.023245 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"de6e1dbd7017e381fdeb008c14987499","Type":"ContainerStarted","Data":"c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6"} Oct 30 21:27:50.049879 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.049762 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"6670d9ca0a34b37e015f8cc739976c92","Type":"ContainerStarted","Data":"f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e"} Oct 30 21:27:50.060497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.060415 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bc4bx" event={"ID":"55b6c776-a4e5-492f-a017-3c9dfc567d56","Type":"ContainerStarted","Data":"3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644"} Oct 30 21:27:50.063565 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.063534 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerStarted","Data":"0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa"} Oct 30 21:27:50.071293 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.071267 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r7866" event={"ID":"9233cc4d-86f9-4890-9464-33c7eefd42a8","Type":"ContainerStarted","Data":"cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54"} Oct 30 21:27:50.078240 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.078216 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mcglb" event={"ID":"9e6b899c-d914-4adb-9a7c-230e359b1497","Type":"ContainerStarted","Data":"f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e"} Oct 30 21:27:50.245386 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.245293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:50.245386 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.245352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:50.245624 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.245519 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:50.245624 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.245583 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:52.24556428 +0000 UTC m=+5.006890895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:50.246023 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.245974 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:50.246205 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.246041 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:52.246008462 +0000 UTC m=+5.007335087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:50.346348 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.346309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:50.346615 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.346596 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:50.346696 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.346623 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:50.346696 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.346637 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:50.346807 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.346697 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:52.346679145 +0000 UTC m=+5.108005760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:50.711484 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.711393 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:22:49 +0000 UTC" deadline="2028-05-11 14:09:34.460000062 +0000 UTC" Oct 30 21:27:50.711484 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.711433 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13408h41m43.748571414s" Oct 30 21:27:50.891142 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.891109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:50.891349 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.891252 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:27:50.891733 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:50.891537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:50.891733 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:50.891687 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:27:51.914272 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:51.914238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:51.914809 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:51.914374 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:27:52.263296 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:52.262405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:52.263296 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:52.262467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:52.263296 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.262627 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:52.263296 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.262688 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:56.262670245 +0000 UTC m=+9.023996860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:52.263296 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.262753 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:52.263296 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.262789 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:56.262776882 +0000 UTC m=+9.024103493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:52.363947 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:52.363202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:52.363947 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.363504 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:52.363947 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.363526 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:52.363947 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.363540 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:52.363947 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.363606 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:56.363586836 +0000 UTC m=+9.124913467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:52.890684 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:52.890580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:52.890854 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.890718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:27:52.891174 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:52.891153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:52.891307 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:52.891272 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:27:53.904875 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:53.904662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:53.905412 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:53.904993 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:27:54.101597 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.101482 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"de6e1dbd7017e381fdeb008c14987499","Type":"ContainerStarted","Data":"ff485c1363d15c4e1713098883e2d9a68e030aff7840656de898156cdd9847b1"} Oct 30 21:27:54.114853 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.114180 2580 generic.go:356] "Generic (PLEG): container finished" podID="6670d9ca0a34b37e015f8cc739976c92" containerID="11f3d88fdb7194f9ddb9e4ba3bcba853f12d2063b26006697cf830ee4d69ed11" exitCode=0 Oct 30 21:27:54.114853 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.114275 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"6670d9ca0a34b37e015f8cc739976c92","Type":"ContainerDied","Data":"11f3d88fdb7194f9ddb9e4ba3bcba853f12d2063b26006697cf830ee4d69ed11"} Oct 30 21:27:54.122586 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.121557 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" podStartSLOduration=5.121540241 podStartE2EDuration="5.121540241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:54.121285225 +0000 UTC m=+6.882611870" watchObservedRunningTime="2026-10-30 21:27:54.121540241 +0000 UTC m=+6.882866877" Oct 30 21:27:54.124801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.123936 2580 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="a786d57ab57f6980a34b5f58f264e0a06b8598d933261893f75c7b12aa697738" exitCode=0 Oct 30 21:27:54.124801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.123978 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerDied","Data":"a786d57ab57f6980a34b5f58f264e0a06b8598d933261893f75c7b12aa697738"} Oct 30 21:27:54.891569 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.890901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:54.891569 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:54.891090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:54.891569 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:54.891172 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:27:54.891569 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:54.891520 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:27:55.136347 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:55.136310 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"6670d9ca0a34b37e015f8cc739976c92","Type":"ContainerStarted","Data":"74c2e8c3f9ec0038149ca4436bf76173d2d83130ad98feeeff134a68f347db0b"} Oct 30 21:27:55.150432 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:55.150378 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mcglb" event={"ID":"9e6b899c-d914-4adb-9a7c-230e359b1497","Type":"ContainerStarted","Data":"3ca2137c9150712f606ffb5e757347b5d09806c3223e0c90b65c797742eaf266"} Oct 30 21:27:55.184087 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:55.184035 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" podStartSLOduration=6.184016083 podStartE2EDuration="6.184016083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:55.160671541 +0000 UTC m=+7.921998176" watchObservedRunningTime="2026-10-30 21:27:55.184016083 +0000 UTC m=+7.945342713" Oct 30 21:27:55.895741 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:55.895257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:55.895741 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:55.895371 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:27:56.154992 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.154872 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" event={"ID":"5dd38803-1d13-470a-97b1-99bc9262d8f0","Type":"ContainerStarted","Data":"42d23bfaab95cb33ad08fb81bf513911f5ac00e6476161facf9e3ccb64b38aad"} Oct 30 21:27:56.158241 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.158196 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8l22v" event={"ID":"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada","Type":"ContainerStarted","Data":"59472c29b0a529b9a17bd9ee18387000801072de5458205877dc19588e68f98f"} Oct 30 21:27:56.174826 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.174778 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mcglb" podStartSLOduration=2.977358051 podStartE2EDuration="8.174762085s" totalImagesPullingTime="5.197404034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:55.184667267 +0000 UTC m=+7.945993897" watchObservedRunningTime="2026-10-30 21:27:56.174762085 +0000 UTC m=+8.936088716" Oct 30 21:27:56.294321 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.294287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:56.294450 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.294387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:56.294535 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.294518 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:56.294594 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.294583 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:04.294565582 +0000 UTC m=+17.055892196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:56.295003 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.294985 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:56.295096 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.295036 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:04.295021582 +0000 UTC m=+17.056348213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:56.395446 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.395414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:56.395769 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.395751 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:56.395846 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.395776 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:56.395846 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.395790 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:56.396025 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.395860 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:04.395842564 +0000 UTC m=+17.157169195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:56.855476 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.855385 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:56.856239 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.856210 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:56.875617 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.875553 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8l22v" podStartSLOduration=2.867621137 podStartE2EDuration="8.875536006s" totalImagesPullingTime="6.007914869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:56.17587663 +0000 UTC m=+8.937203259" watchObservedRunningTime="2026-10-30 21:27:56.875536006 +0000 UTC m=+9.636862639" Oct 30 21:27:56.891125 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.891098 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:56.891336 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.891275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:27:56.891410 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:56.891357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:56.891693 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:56.891476 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:27:57.161978 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.161727 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:57.162423 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.162313 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:27:57.686812 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.686753 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:27:57.693464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.693046 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:27:57.686769067Z","UUID":"c24191cc-fe53-482e-b071-a3619c0e9933","Handler":null,"Name":"","Endpoint":""} Oct 30 21:27:57.694341 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.694325 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:27:57.694438 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.694350 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:27:57.898223 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:57.898168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:57.898373 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:57.898329 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:27:58.165060 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.164984 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r7866" event={"ID":"9233cc4d-86f9-4890-9464-33c7eefd42a8","Type":"ContainerStarted","Data":"1d10fef25a1d641bd2b914698e33d5f63adf39045cf72dca6476c65b2ae76ab1"} Oct 30 21:27:58.167413 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.167354 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" event={"ID":"5dd38803-1d13-470a-97b1-99bc9262d8f0","Type":"ContainerStarted","Data":"93b9843031b7cc8e884cff05f611e81f53024f65cfc1cd946dc434cf16d524ec"} Oct 30 21:27:58.169681 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.169656 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nx794" event={"ID":"4422cee8-aec1-4569-9277-bba73b00a9ad","Type":"ContainerStarted","Data":"59feb02a74ac33d96b846b56424136b3ee8d7d29d4335808e4eb3e279bf462d9"} Oct 30 21:27:58.206144 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.205920 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r7866" podStartSLOduration=1.441994825 podStartE2EDuration="10.205903587s" totalImagesPullingTime="8.763908762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:58.184064283 +0000 UTC m=+10.945390913" watchObservedRunningTime="2026-10-30 21:27:58.205903587 +0000 UTC m=+10.967230217" Oct 30 21:27:58.206267 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.206173 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nx794" podStartSLOduration=1.631628592 podStartE2EDuration="10.206165335s" totalImagesPullingTime="8.574536743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:58.205315191 +0000 UTC m=+10.966641825" watchObservedRunningTime="2026-10-30 21:27:58.206165335 +0000 UTC m=+10.967491944" Oct 30 21:27:58.890795 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.890555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:27:58.890984 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:58.890563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:27:58.890984 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:58.890869 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:27:58.890984 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:58.890971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:27:59.174043 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:59.173892 2580 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="50f26ec6e19d9bdb4d877116168005e73f426389b40dd7fb73e48a800d98e994" exitCode=0 Oct 30 21:27:59.174529 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:59.174199 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerDied","Data":"50f26ec6e19d9bdb4d877116168005e73f426389b40dd7fb73e48a800d98e994"} Oct 30 21:27:59.177792 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:59.177767 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" event={"ID":"5dd38803-1d13-470a-97b1-99bc9262d8f0","Type":"ContainerStarted","Data":"14cab8e60c7d3aec77e37adc18d7ae23ce9fbaaa5944901e21b18b2085ca69c5"} Oct 30 21:27:59.228428 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:59.228365 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" podStartSLOduration=2.3161984 podStartE2EDuration="11.228347936s" totalImagesPullingTime="8.912149536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:59.22738181 +0000 UTC m=+11.988708440" watchObservedRunningTime="2026-10-30 21:27:59.228347936 +0000 UTC m=+11.989674564" Oct 30 21:27:59.890577 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:27:59.890550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:27:59.890751 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:27:59.890657 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:00.182299 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:00.181714 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bc4bx" event={"ID":"55b6c776-a4e5-492f-a017-3c9dfc567d56","Type":"ContainerStarted","Data":"05ba8dd7b708019e3d8f5fcf6bbc7f9d7f038b3307159647c83975a53e66fe1c"} Oct 30 21:28:00.200795 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:00.200750 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bc4bx" podStartSLOduration=3.37460058 podStartE2EDuration="12.200737754s" totalImagesPullingTime="8.826137174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:00.200195907 +0000 UTC m=+12.961522535" watchObservedRunningTime="2026-10-30 21:28:00.200737754 +0000 UTC m=+12.962064382" Oct 30 21:28:00.891177 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:00.891147 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:00.891342 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:00.891267 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:00.891342 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:00.891318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:00.891470 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:00.891430 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:01.186185 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:01.186093 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" event={"ID":"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4","Type":"ContainerStarted","Data":"0e791cba1b2c22ec1958c8a2593ef2703fb3113d1deed9c971d454fca3fdf8fc"} Oct 30 21:28:01.188433 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:01.188406 2580 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="472e936680e595d8dc89b54f2cfaa7ec3bae4cb90f5d3782f41d130a3d74afae" exitCode=0 Oct 30 21:28:01.188546 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:01.188493 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerDied","Data":"472e936680e595d8dc89b54f2cfaa7ec3bae4cb90f5d3782f41d130a3d74afae"} Oct 30 21:28:01.207545 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:01.207490 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" podStartSLOduration=1.9879062109999999 podStartE2EDuration="13.207472299s" totalImagesPullingTime="11.219566088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:01.206258935 +0000 UTC m=+13.967585568" watchObservedRunningTime="2026-10-30 21:28:01.207472299 +0000 UTC m=+13.968798929" Oct 30 21:28:01.890718 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:01.890686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:01.890888 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:01.890816 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:02.193201 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:02.193114 2580 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="10174efb224d79369736b5cf7545bd046c9cf8c41bac0acd96a349d7fe8f5958" exitCode=0 Oct 30 21:28:02.193201 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:02.193153 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerDied","Data":"10174efb224d79369736b5cf7545bd046c9cf8c41bac0acd96a349d7fe8f5958"} Oct 30 21:28:02.890872 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:02.890659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:02.891063 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:02.890984 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:02.891321 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:02.891306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:02.891426 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:02.891410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:03.890683 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:03.890650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:03.891152 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:03.890770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:04.355987 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:04.355941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:04.356100 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:04.356020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:04.356100 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.356063 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:28:04.356164 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.356115 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:28:04.356164 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.356129 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:20.356112312 +0000 UTC m=+33.117438934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:28:04.356240 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.356167 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:20.356149774 +0000 UTC m=+33.117476395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:28:04.457209 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:04.457159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:04.457304 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.457291 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:28:04.457338 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.457307 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:28:04.457338 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.457317 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:28:04.457404 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.457356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:20.457345029 +0000 UTC m=+33.218671637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:28:04.890531 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:04.890501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:04.890645 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.890603 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:04.890645 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:04.890626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:04.890749 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:04.890673 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:05.202447 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.202288 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"4d42e8f8175aabc34a187633be2fc0d6c9dec5070e1cce7aeaacf3ba1ea1bf3e"} Oct 30 21:28:05.202624 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.202530 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"8e31b69af608491324744590d68e0dfa59404b70295a01facb5a4fc8ef24ec5b"} Oct 30 21:28:05.202624 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.202586 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"326403b1574cf646cd5a879a72f4db5556b922d590b1c22a18c1941a7c900d87"} Oct 30 21:28:05.202711 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.202644 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"1a52f413b223d9b38a181c3847801115e3a26d0bc419a0c267988f979a3b6b40"} Oct 30 21:28:05.202711 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.202690 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"0a369e7a49068a7235fdc73d132e1ea08093b7e547edd7939dbb872ba5dfc807"} Oct 30 21:28:05.202782 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.202731 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"d98d3fb905c3f6e2ecf985fa6de4c6bf97f09469a79dc98f200caefa3cd7ad6f"} Oct 30 21:28:05.205111 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.205083 2580 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="82f8d0dfad86ceab135e4ede538c7d08b1dafed3ff92bf9bb9fa0fb9006bdcbf" exitCode=0 Oct 30 21:28:05.205203 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.205132 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerDied","Data":"82f8d0dfad86ceab135e4ede538c7d08b1dafed3ff92bf9bb9fa0fb9006bdcbf"} Oct 30 21:28:05.792123 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.792100 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7866_9233cc4d-86f9-4890-9464-33c7eefd42a8/dns-node-resolver/0.log" Oct 30 21:28:05.890502 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:05.890474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:05.890649 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:05.890563 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:06.210209 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:06.210137 2580 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="6c4587e6fabcd380aeefd0d9523c77d4a62e4c868015f38d2e07b09bf2d26d2e" exitCode=0 Oct 30 21:28:06.210209 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:06.210171 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerDied","Data":"6c4587e6fabcd380aeefd0d9523c77d4a62e4c868015f38d2e07b09bf2d26d2e"} Oct 30 21:28:06.340019 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:06.339996 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8l22v_40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada/node-ca/0.log" Oct 30 21:28:06.890443 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:06.890420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:06.890574 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:06.890510 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:06.890574 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:06.890565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:06.890663 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:06.890645 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:07.216203 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:07.216167 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"513936ef9bf8102de729c1fa2ec7e017b6427b5c0178dea6f5b2bca02970a832"} Oct 30 21:28:07.219395 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:07.219372 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87ql4" event={"ID":"89806e93-18c5-43a5-b1ed-0f6eae4f20f3","Type":"ContainerStarted","Data":"4f801a7877d0dd743775d369e68d3a379ed7eded89048bcd1af2b34d4a4db9f5"} Oct 30 21:28:07.247502 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:07.247459 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-87ql4" podStartSLOduration=7.964265689 podStartE2EDuration="19.2474476s" totalImagesPullingTime="11.283181911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:07.247250733 +0000 UTC m=+20.008577362" watchObservedRunningTime="2026-10-30 21:28:07.2474476 +0000 UTC m=+20.008774229" Oct 30 21:28:07.892334 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:07.892304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:07.892494 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:07.892413 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:08.890506 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:08.890472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:08.891169 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:08.890596 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:08.891169 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:08.890642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:08.891169 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:08.890731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:09.890876 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:09.890701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:09.891421 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:09.890978 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:10.228464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:10.228392 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" event={"ID":"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f","Type":"ContainerStarted","Data":"a78074abf2204420e1ef52f917c88aed5581bf8dfe26608a504f495363219504"} Oct 30 21:28:10.228663 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:10.228645 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:10.244033 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:10.244010 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:10.304678 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:10.304626 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" podStartSLOduration=6.519224048 podStartE2EDuration="22.304611694s" totalImagesPullingTime="15.785387646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:10.268160766 +0000 UTC m=+23.029487396" watchObservedRunningTime="2026-10-30 21:28:10.304611694 +0000 UTC m=+23.065938324" Oct 30 21:28:10.890471 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:10.890447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:10.890691 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:10.890566 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:10.890691 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:10.890601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:10.890691 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:10.890668 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:11.230420 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:11.230365 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:28:11.230785 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:11.230739 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:11.245104 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:11.245085 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:11.891051 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:11.891014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:11.891327 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:11.891127 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:12.232534 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:12.232463 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:28:12.890348 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:12.890313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:12.890517 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:12.890425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:12.890517 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:12.890476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:12.890592 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:12.890573 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:13.234463 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:13.234395 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:28:13.890751 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:13.890720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:13.890928 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:13.890836 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:13.890928 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:13.890891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:13.891036 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:13.890996 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:14.890316 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:14.890292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:14.890700 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:14.890381 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:15.892944 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:15.892913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:15.893429 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:15.893092 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:15.893429 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:15.893127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:15.893429 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:15.893224 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:16.890475 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:16.890447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:16.890620 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:16.890538 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:17.893308 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:17.893282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:17.893717 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:17.893391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:17.893717 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:17.893444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:17.893717 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:17.893521 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:18.254179 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:18.254100 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:18.254315 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:18.254287 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:28:18.267424 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:18.267391 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" podUID="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 21:28:18.275460 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:18.275433 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" podUID="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 21:28:18.890902 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:18.890878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:18.891066 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:18.890975 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:19.891163 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:19.891131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:19.891590 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:19.891235 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:19.891590 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:19.891322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:19.891590 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:19.891437 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:20.374159 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:20.374092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:20.374159 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:20.374126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:20.374346 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.374232 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:28:20.374346 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.374244 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:28:20.374346 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.374288 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret podName:5f8be18f-c1ce-46ae-b36c-df83e438c101 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:52.374276142 +0000 UTC m=+65.135602749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret") pod "global-pull-secret-syncer-mq6jf" (UID: "5f8be18f-c1ce-46ae-b36c-df83e438c101") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:28:20.374346 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.374302 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs podName:75908611-03b2-42df-85e5-128bd14c5a43 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:52.374296073 +0000 UTC m=+65.135622680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs") pod "network-metrics-daemon-p2nkx" (UID: "75908611-03b2-42df-85e5-128bd14c5a43") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:28:20.475132 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:20.475108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:20.475246 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.475225 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:28:20.475246 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.475237 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:28:20.475246 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.475246 2580 projected.go:196] Error preparing data for projected volume kube-api-access-d5j5r for pod openshift-network-diagnostics/network-check-target-sq9rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:28:20.475361 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.475283 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r podName:09c3c33e-a59d-4ae8-abec-366b2effbb07 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:52.47527303 +0000 UTC m=+65.236599638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5j5r" (UniqueName: "kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r") pod "network-check-target-sq9rx" (UID: "09c3c33e-a59d-4ae8-abec-366b2effbb07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:28:20.891199 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:20.891177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:20.891549 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:20.891264 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:21.891154 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:21.891129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:21.891305 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:21.891135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:21.891305 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:21.891224 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:21.891305 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:21.891292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:22.890677 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:22.890649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:22.890849 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:22.890754 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:23.890583 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:23.890556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:23.890972 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:23.890556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:23.890972 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:23.890658 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:23.890972 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:23.890710 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:24.890550 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:24.890520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:24.890701 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:24.890606 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:25.891157 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:25.891123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:25.891630 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:25.891307 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:25.891630 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:25.891359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:25.891630 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:25.891486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:26.147296 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:26.147243 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mq6jf"] Oct 30 21:28:26.148484 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:26.148330 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2nkx"] Oct 30 21:28:26.151589 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:26.151566 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sq9rx"] Oct 30 21:28:26.151691 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:26.151679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:26.151807 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:26.151781 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:26.257609 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:26.257590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:26.257718 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:26.257590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:26.257754 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:26.257710 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:26.257790 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:26.257768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:27.892961 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:27.892786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:27.893298 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:27.892881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:27.893298 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:27.893029 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:27.893298 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:27.892929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:27.893298 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:27.893104 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:27.893298 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:27.893168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:29.890705 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:29.890675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:29.891156 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:29.890685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:29.891156 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:29.890810 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sq9rx" podUID="09c3c33e-a59d-4ae8-abec-366b2effbb07" Oct 30 21:28:29.891156 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:29.890831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:29.891644 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:29.891611 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2nkx" podUID="75908611-03b2-42df-85e5-128bd14c5a43" Oct 30 21:28:29.891772 ip-10-0-137-168 kubenswrapper[2580]: E1030 21:28:29.891753 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq6jf" podUID="5f8be18f-c1ce-46ae-b36c-df83e438c101" Oct 30 21:28:30.486672 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.486608 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeReady" Oct 30 21:28:30.486793 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.486718 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:28:30.540326 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.540303 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xfdq8"] Oct 30 21:28:30.542092 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.542076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.544260 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.544241 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8btfx"] Oct 30 21:28:30.545772 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.545756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.560230 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.560213 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:28:30.560230 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.560221 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:28:30.564719 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.564700 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcj4q\"" Oct 30 21:28:30.565597 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.565577 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfdq8"] Oct 30 21:28:30.565905 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.565884 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:28:30.566039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.566011 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:28:30.566157 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.566141 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:28:30.566435 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.566413 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8btfx"] Oct 30 21:28:30.566788 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.566768 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r96r5\"" Oct 30 21:28:30.582039 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.582020 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8lm4w"] Oct 30 21:28:30.583682 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.583667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.642460 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.642438 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8lm4w"] Oct 30 21:28:30.647850 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.647830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d852c03-e50d-4039-bd52-7bfae081a271-config-volume\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.647927 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.647859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trx6c\" (UniqueName: \"kubernetes.io/projected/a9f17f5b-18dc-4642-a2bb-d778b0524561-kube-api-access-trx6c\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.647989 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.647975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8jcz\" (UniqueName: \"kubernetes.io/projected/8d852c03-e50d-4039-bd52-7bfae081a271-kube-api-access-r8jcz\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.648028 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.647997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f17f5b-18dc-4642-a2bb-d778b0524561-cert\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.648028 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.648023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d852c03-e50d-4039-bd52-7bfae081a271-metrics-tls\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.648092 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.648040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d852c03-e50d-4039-bd52-7bfae081a271-tmp-dir\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.656731 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.656717 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:28:30.656788 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.656771 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:28:30.656956 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.656932 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:28:30.657943 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.657929 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:28:30.658003 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.657989 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wf4q\"" Oct 30 21:28:30.749182 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d852c03-e50d-4039-bd52-7bfae081a271-config-volume\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.749182 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trx6c\" (UniqueName: \"kubernetes.io/projected/a9f17f5b-18dc-4642-a2bb-d778b0524561-kube-api-access-trx6c\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.749182 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4693396-79bd-410d-86a3-5a61f614d231-data-volume\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.749330 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4693396-79bd-410d-86a3-5a61f614d231-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.749330 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88c2z\" (UniqueName: \"kubernetes.io/projected/c4693396-79bd-410d-86a3-5a61f614d231-kube-api-access-88c2z\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.749392 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jcz\" (UniqueName: \"kubernetes.io/projected/8d852c03-e50d-4039-bd52-7bfae081a271-kube-api-access-r8jcz\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.749392 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f17f5b-18dc-4642-a2bb-d778b0524561-cert\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.749392 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4693396-79bd-410d-86a3-5a61f614d231-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.749507 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d852c03-e50d-4039-bd52-7bfae081a271-metrics-tls\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.749507 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d852c03-e50d-4039-bd52-7bfae081a271-tmp-dir\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.749507 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749483 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4693396-79bd-410d-86a3-5a61f614d231-crio-socket\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.749746 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749729 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d852c03-e50d-4039-bd52-7bfae081a271-tmp-dir\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.749941 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.749924 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d852c03-e50d-4039-bd52-7bfae081a271-config-volume\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.753158 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.753139 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f17f5b-18dc-4642-a2bb-d778b0524561-cert\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.753238 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.753160 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d852c03-e50d-4039-bd52-7bfae081a271-metrics-tls\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.762281 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.762227 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jcz\" (UniqueName: \"kubernetes.io/projected/8d852c03-e50d-4039-bd52-7bfae081a271-kube-api-access-r8jcz\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.762435 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.762421 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trx6c\" (UniqueName: \"kubernetes.io/projected/a9f17f5b-18dc-4642-a2bb-d778b0524561-kube-api-access-trx6c\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.849884 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.849861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4693396-79bd-410d-86a3-5a61f614d231-crio-socket\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.849984 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.849890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4693396-79bd-410d-86a3-5a61f614d231-data-volume\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.849984 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.849907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4693396-79bd-410d-86a3-5a61f614d231-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.849984 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.849927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88c2z\" (UniqueName: \"kubernetes.io/projected/c4693396-79bd-410d-86a3-5a61f614d231-kube-api-access-88c2z\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.850136 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.849998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4693396-79bd-410d-86a3-5a61f614d231-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.850136 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.850094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4693396-79bd-410d-86a3-5a61f614d231-crio-socket\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.850241 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.850218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4693396-79bd-410d-86a3-5a61f614d231-data-volume\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.850528 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.850501 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4693396-79bd-410d-86a3-5a61f614d231-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.852033 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.852015 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4693396-79bd-410d-86a3-5a61f614d231-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.854915 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.854898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:30.858054 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.858040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:30.863045 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.862983 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88c2z\" (UniqueName: \"kubernetes.io/projected/c4693396-79bd-410d-86a3-5a61f614d231-kube-api-access-88c2z\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:30.895176 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:30.894370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:31.074777 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.074715 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7866_9233cc4d-86f9-4890-9464-33c7eefd42a8/dns-node-resolver/0.log" Oct 30 21:28:31.139678 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.139655 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8btfx"] Oct 30 21:28:31.142511 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.142490 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfdq8"] Oct 30 21:28:31.145550 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:31.145498 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d852c03_e50d_4039_bd52_7bfae081a271.slice/crio-9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b WatchSource:0}: Error finding container 9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b: Status 404 returned error can't find the container with id 9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b Oct 30 21:28:31.190215 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.190087 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8lm4w"] Oct 30 21:28:31.193395 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:31.193370 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4693396_79bd_410d_86a3_5a61f614d231.slice/crio-4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c WatchSource:0}: Error finding container 4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c: Status 404 returned error can't find the container with id 4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c Oct 30 21:28:31.268749 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.268720 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lm4w" event={"ID":"c4693396-79bd-410d-86a3-5a61f614d231","Type":"ContainerStarted","Data":"1cbe448e3fc76f1d05fefae7bc054eec547219a2d45b36544bc359877b23524d"} Oct 30 21:28:31.268859 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.268802 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lm4w" event={"ID":"c4693396-79bd-410d-86a3-5a61f614d231","Type":"ContainerStarted","Data":"4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c"} Oct 30 21:28:31.269995 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.269935 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8btfx" event={"ID":"a9f17f5b-18dc-4642-a2bb-d778b0524561","Type":"ContainerStarted","Data":"43d97fae832e343d1246c7ac616ce94a2c953740df2c4936bd32c18716121a5d"} Oct 30 21:28:31.271074 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.271056 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfdq8" event={"ID":"8d852c03-e50d-4039-bd52-7bfae081a271","Type":"ContainerStarted","Data":"9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b"} Oct 30 21:28:31.875432 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.874078 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8l22v_40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada/node-ca/0.log" Oct 30 21:28:31.896363 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.895290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:31.896363 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.895579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:31.896363 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.895867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:31.902637 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.902616 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpdb2\"" Oct 30 21:28:31.902762 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.902739 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:28:31.905608 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.905589 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:28:31.906315 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.906295 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjjj4\"" Oct 30 21:28:31.906473 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.906455 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4rkh\"" Oct 30 21:28:31.915504 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.915484 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:28:31.915599 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:31.915532 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:28:32.223790 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.222772 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z"] Oct 30 21:28:32.225030 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.225009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.233783 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.233761 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5jcj9\"" Oct 30 21:28:32.234643 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.233892 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:28:32.234643 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.234008 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:28:32.234643 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.234052 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:28:32.234643 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.234167 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:28:32.235177 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.235159 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:28:32.237817 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.237735 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z"] Oct 30 21:28:32.253409 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.253388 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8psws"] Oct 30 21:28:32.255596 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.255577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.264773 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.264752 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf"] Oct 30 21:28:32.267023 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.267003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.276914 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.276885 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lm4w" event={"ID":"c4693396-79bd-410d-86a3-5a61f614d231","Type":"ContainerStarted","Data":"242272e7390f92320cc3a84a5a083a783373f5f721def29b102ab3e5279d419b"} Oct 30 21:28:32.281583 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.281560 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf"] Oct 30 21:28:32.328244 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.328212 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:28:32.328464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.328441 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:28:32.328589 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.328498 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z4fj8\"" Oct 30 21:28:32.328589 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.328547 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:28:32.328589 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.328572 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:28:32.328708 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.328597 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9vdcb\"" Oct 30 21:28:32.333552 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.333520 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:28:32.335518 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.335493 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:28:32.372315 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckqp\" (UniqueName: \"kubernetes.io/projected/136a5087-3017-46af-aa81-15b1a899031a-kube-api-access-wckqp\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.372434 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-metrics-client-ca\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.372434 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.372519 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-tls\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.372519 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/136a5087-3017-46af-aa81-15b1a899031a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.372519 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxz9l\" (UniqueName: \"kubernetes.io/projected/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-kube-api-access-xxz9l\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.372519 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-sys\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.372519 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-root\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.372716 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bsw\" (UniqueName: \"kubernetes.io/projected/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-kube-api-access-82bsw\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.372716 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.372716 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.372919 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.372919 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.372919 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-textfile\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.373170 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-wtmp\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.373170 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.372989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.373170 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.373052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.373170 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.373095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.373170 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.373154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.474559 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckqp\" (UniqueName: \"kubernetes.io/projected/136a5087-3017-46af-aa81-15b1a899031a-kube-api-access-wckqp\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.474559 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-metrics-client-ca\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.474559 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-tls\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/136a5087-3017-46af-aa81-15b1a899031a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxz9l\" (UniqueName: \"kubernetes.io/projected/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-kube-api-access-xxz9l\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-sys\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-root\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bsw\" (UniqueName: \"kubernetes.io/projected/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-kube-api-access-82bsw\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.474801 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-textfile\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-wtmp\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.474980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.475008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.475148 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.475037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.475588 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.475559 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-metrics-client-ca\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.475913 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.475890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/136a5087-3017-46af-aa81-15b1a899031a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.476452 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.476433 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.476511 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.476496 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-sys\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.476551 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.476536 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-root\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.478065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.477236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-wtmp\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.478065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.477500 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-textfile\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.478065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.477756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.478065 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.478013 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.478303 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.478255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.479289 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.479243 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-tls\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.479657 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.479636 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.480651 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.480630 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.481302 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.481276 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.481515 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.481492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.486810 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.486782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.489161 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.488930 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bsw\" (UniqueName: \"kubernetes.io/projected/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-kube-api-access-82bsw\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.489161 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.489102 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckqp\" (UniqueName: \"kubernetes.io/projected/136a5087-3017-46af-aa81-15b1a899031a-kube-api-access-wckqp\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.489413 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.489252 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxz9l\" (UniqueName: \"kubernetes.io/projected/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-kube-api-access-xxz9l\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.540879 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.540657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:32.579387 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.579351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:32.582069 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.582041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:32.827713 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.827431 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z"] Oct 30 21:28:32.833046 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:32.833015 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdaf21b_e4a6_4c90_8eb2_4dbf75dfa59f.slice/crio-e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf WatchSource:0}: Error finding container e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf: Status 404 returned error can't find the container with id e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf Oct 30 21:28:32.874132 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:32.874106 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf"] Oct 30 21:28:32.878503 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:32.878475 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136a5087_3017_46af_aa81_15b1a899031a.slice/crio-65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1 WatchSource:0}: Error finding container 65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1: Status 404 returned error can't find the container with id 65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1 Oct 30 21:28:33.282213 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.282136 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8psws" event={"ID":"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9","Type":"ContainerStarted","Data":"0ee47fc53bc4db11db3c00a0bb3be2f0658cb61a4aa06cbbb5e1e3c7ce9b7182"} Oct 30 21:28:33.284173 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.284143 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" event={"ID":"136a5087-3017-46af-aa81-15b1a899031a","Type":"ContainerStarted","Data":"65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1"} Oct 30 21:28:33.288478 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.288447 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" event={"ID":"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f","Type":"ContainerStarted","Data":"3b46ee3cb2b096be12de0353c5f6365edc9f0f8c53cf293de0032910610eb542"} Oct 30 21:28:33.288587 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.288507 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" event={"ID":"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f","Type":"ContainerStarted","Data":"f6525a8f89cdd114def51ae215019b3dcd168ad32823d85b21a7f81255450980"} Oct 30 21:28:33.288587 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.288540 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" event={"ID":"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f","Type":"ContainerStarted","Data":"e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf"} Oct 30 21:28:33.333045 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.333016 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:28:33.336461 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.336440 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.343980 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.343880 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:28:33.344254 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.344143 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:28:33.344254 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.344217 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:28:33.344387 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.344282 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:28:33.344387 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.344318 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:28:33.344387 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.344317 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:28:33.345638 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.345596 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:28:33.345729 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.345711 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:28:33.345815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.345802 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-48dkn\"" Oct 30 21:28:33.351272 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.351238 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:28:33.370788 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.370720 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.486989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.487665 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.487415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.589863 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.588767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.595480 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.594515 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.595933 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.595604 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.597492 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.596799 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.597492 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.597448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.600409 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.597860 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.602215 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.602120 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.602902 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.602800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.604837 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.604813 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.606213 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.606011 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.606213 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.606152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.606403 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.606379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.608041 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.608011 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.608406 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.608385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:33.673938 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:33.673907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:34.019320 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.017851 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:28:34.131203 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.131170 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-546ff4d698-mbksn"] Oct 30 21:28:34.133891 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.133870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.151314 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151261 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:28:34.151314 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151310 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:28:34.151497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151329 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:28:34.151497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151365 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:28:34.151497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151394 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:28:34.151497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151445 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:28:34.151497 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151488 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:28:34.151672 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.151554 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dj9gj\"" Oct 30 21:28:34.157179 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.157150 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:28:34.157710 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.157682 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546ff4d698-mbksn"] Oct 30 21:28:34.254274 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.254245 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f995ff688-xs5n7"] Oct 30 21:28:34.257597 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.257579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.269967 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.269932 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:28:34.270088 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.270075 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:28:34.270203 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.270190 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mgk6x\"" Oct 30 21:28:34.270667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.270266 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:28:34.270667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.270352 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:28:34.270667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.270405 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:28:34.270667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.270493 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aahe206i1kld1\"" Oct 30 21:28:34.274973 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.274939 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f995ff688-xs5n7"] Oct 30 21:28:34.295118 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.294406 2580 generic.go:356] "Generic (PLEG): container finished" podID="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" containerID="126480897b8c3ebe863aa9f0fec1f0ee88142f744755c76cfb646fb444bbb435" exitCode=0 Oct 30 21:28:34.295118 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.294532 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8psws" event={"ID":"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9","Type":"ContainerDied","Data":"126480897b8c3ebe863aa9f0fec1f0ee88142f744755c76cfb646fb444bbb435"} Oct 30 21:28:34.296646 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.296762 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.296762 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.296762 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.296916 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.296916 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.296916 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.297083 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.296914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.298108 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.298079 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfdq8" event={"ID":"8d852c03-e50d-4039-bd52-7bfae081a271","Type":"ContainerStarted","Data":"6c89642a7df0ec252e43a3b37dae2c1bff43614e28fd811381fb2fc2f8514f7b"} Oct 30 21:28:34.298200 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.298131 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfdq8" event={"ID":"8d852c03-e50d-4039-bd52-7bfae081a271","Type":"ContainerStarted","Data":"866bebed009fc333872f5a0b135750413cdc7189fe25d729bd77d8c27a39fd5e"} Oct 30 21:28:34.298200 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.298194 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:34.299828 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.299805 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"32a94c505a332ec8116af313fa49d2586dc06556412c624074ee777aa7f279be"} Oct 30 21:28:34.329502 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.329454 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xfdq8" podStartSLOduration=2.210367168 podStartE2EDuration="4.329436209s" totalImagesPullingTime="2.119069041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:34.328410916 +0000 UTC m=+47.089737547" watchObservedRunningTime="2026-10-30 21:28:34.329436209 +0000 UTC m=+47.090762840" Oct 30 21:28:34.397760 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397760 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.397929 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4e22dd8-f90b-4512-9ade-3f3ea3329946-metrics-client-ca\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.397998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-grpc-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e22dd8-f90b-4512-9ade-3f3ea3329946-tmp\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398512 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gck7r\" (UniqueName: \"kubernetes.io/projected/b4e22dd8-f90b-4512-9ade-3f3ea3329946-kube-api-access-gck7r\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.398870 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.398839 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.402082 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.399309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.402082 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.400488 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.402082 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.401777 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.403238 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.402911 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.404563 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.404524 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.406650 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.406608 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.412430 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.412396 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.491187 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.491151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.498886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4e22dd8-f90b-4512-9ade-3f3ea3329946-metrics-client-ca\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.498930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.498980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-grpc-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e22dd8-f90b-4512-9ade-3f3ea3329946-tmp\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gck7r\" (UniqueName: \"kubernetes.io/projected/b4e22dd8-f90b-4512-9ade-3f3ea3329946-kube-api-access-gck7r\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499464 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.499979 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499668 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4e22dd8-f90b-4512-9ade-3f3ea3329946-metrics-client-ca\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.500698 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.499987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e22dd8-f90b-4512-9ade-3f3ea3329946-tmp\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.502581 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.502525 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.503548 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.503348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-grpc-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.504084 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.504022 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.504457 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.504438 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.505133 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.505110 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.505406 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.505384 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.514096 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.514054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gck7r\" (UniqueName: \"kubernetes.io/projected/b4e22dd8-f90b-4512-9ade-3f3ea3329946-kube-api-access-gck7r\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.583646 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.583614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:34.776311 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.776288 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546ff4d698-mbksn"] Oct 30 21:28:34.778898 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:34.778796 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3231e522_a00c_4ce7_9285_fe3954981036.slice/crio-75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656 WatchSource:0}: Error finding container 75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656: Status 404 returned error can't find the container with id 75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656 Oct 30 21:28:34.873657 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:34.873545 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f995ff688-xs5n7"] Oct 30 21:28:34.886033 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:34.885860 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e22dd8_f90b_4512_9ade_3f3ea3329946.slice/crio-d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19 WatchSource:0}: Error finding container d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19: Status 404 returned error can't find the container with id d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19 Oct 30 21:28:35.306862 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.306828 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" event={"ID":"136a5087-3017-46af-aa81-15b1a899031a","Type":"ContainerStarted","Data":"cd1b4e998f68dfcacc03a7649d43bbacfde79c95426e87e7484163117ef0d640"} Oct 30 21:28:35.307271 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.306895 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" event={"ID":"136a5087-3017-46af-aa81-15b1a899031a","Type":"ContainerStarted","Data":"d6d0a70682fc2bf2e57a8c7edf2aa3d2936fe498ef550421ebf721f9efc49cff"} Oct 30 21:28:35.307271 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.306926 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" event={"ID":"136a5087-3017-46af-aa81-15b1a899031a","Type":"ContainerStarted","Data":"51c7ae772e2fcaec80c48d1ffc127045569edfa5b549d24c1d8c3c883b680676"} Oct 30 21:28:35.308964 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.308927 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" event={"ID":"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f","Type":"ContainerStarted","Data":"c191f432a9d3bb2ce581aabbc14c638bcadb78bf1cf43aec8faf7c533003277c"} Oct 30 21:28:35.310219 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.310200 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19"} Oct 30 21:28:35.311514 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.311486 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ff4d698-mbksn" event={"ID":"3231e522-a00c-4ce7-9285-fe3954981036","Type":"ContainerStarted","Data":"75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656"} Oct 30 21:28:35.313098 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.313078 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4" exitCode=0 Oct 30 21:28:35.313191 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.313162 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4"} Oct 30 21:28:35.318098 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.315358 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8psws" event={"ID":"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9","Type":"ContainerStarted","Data":"4fc991771a70f458b7afa54e506a502efe18eab16cb69cf334c381c354db5933"} Oct 30 21:28:35.318098 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.315396 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8psws" event={"ID":"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9","Type":"ContainerStarted","Data":"25005f1b51f02bf45177c1985b999a989eccd28412f3530308b08ed28b4446b3"} Oct 30 21:28:35.318098 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.317321 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lm4w" event={"ID":"c4693396-79bd-410d-86a3-5a61f614d231","Type":"ContainerStarted","Data":"4585192c5ddd8345615801b29cb8103de58f06f9a84f1a025895b953346a8452"} Oct 30 21:28:35.318820 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.318801 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8btfx" event={"ID":"a9f17f5b-18dc-4642-a2bb-d778b0524561","Type":"ContainerStarted","Data":"d9f110367081d91d2963f5c34297e9ec013189b1a02a105ac8aa83edcc973eb0"} Oct 30 21:28:35.328989 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.328326 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" podStartSLOduration=1.428744531 podStartE2EDuration="3.328312968s" totalImagesPullingTime="1.899568437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.327679167 +0000 UTC m=+48.089005795" watchObservedRunningTime="2026-10-30 21:28:35.328312968 +0000 UTC m=+48.089639597" Oct 30 21:28:35.349361 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.349306 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8btfx" podStartSLOduration=1.746107899 podStartE2EDuration="5.349296736s" totalImagesPullingTime="3.603188837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.348416479 +0000 UTC m=+48.109743110" watchObservedRunningTime="2026-10-30 21:28:35.349296736 +0000 UTC m=+48.110623365" Oct 30 21:28:35.379923 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.379869 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8psws" podStartSLOduration=1.9331707200000001 podStartE2EDuration="3.379851056s" totalImagesPullingTime="1.366552036s" totalInitContainerRuntime="80.1283ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.379091689 +0000 UTC m=+48.140418321" watchObservedRunningTime="2026-10-30 21:28:35.379851056 +0000 UTC m=+48.141177687" Oct 30 21:28:35.413478 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.412971 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" podStartSLOduration=2.108885888 podStartE2EDuration="3.412940629s" totalImagesPullingTime="1.304054741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.412540358 +0000 UTC m=+48.173866991" watchObservedRunningTime="2026-10-30 21:28:35.412940629 +0000 UTC m=+48.174267262" Oct 30 21:28:35.478229 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:35.478145 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8lm4w" podStartSLOduration=2.5566983 podStartE2EDuration="5.478131045s" totalImagesPullingTime="2.921432745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.476817353 +0000 UTC m=+48.238143987" watchObservedRunningTime="2026-10-30 21:28:35.478131045 +0000 UTC m=+48.239457673" Oct 30 21:28:36.663059 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.663021 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bdd5667d8-qdt5r"] Oct 30 21:28:36.665636 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.665586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.674983 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.674961 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:28:36.675122 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.675105 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:28:36.693487 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.693461 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:28:36.693588 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.693501 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ek4vn38sae3e9\"" Oct 30 21:28:36.693588 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.693557 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:28:36.693588 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.693564 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d2jh8\"" Oct 30 21:28:36.693815 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.693769 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bdd5667d8-qdt5r"] Oct 30 21:28:36.822260 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-client-certs\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.822260 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-tls\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.822504 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpw5\" (UniqueName: \"kubernetes.io/projected/0328b8ce-272d-4232-8681-ef9e0d3a5274-kube-api-access-gcpw5\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.822571 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-metrics-server-audit-profiles\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.822571 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-client-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.822701 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0328b8ce-272d-4232-8681-ef9e0d3a5274-audit-log\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.822766 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.822717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.923992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0328b8ce-272d-4232-8681-ef9e0d3a5274-audit-log\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.924039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.924083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-client-certs\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.924107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-tls\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.924136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpw5\" (UniqueName: \"kubernetes.io/projected/0328b8ce-272d-4232-8681-ef9e0d3a5274-kube-api-access-gcpw5\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.924226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-metrics-server-audit-profiles\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.924667 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.924265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-client-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.927569 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.926590 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-metrics-server-audit-profiles\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.927569 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.926874 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0328b8ce-272d-4232-8681-ef9e0d3a5274-audit-log\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.927569 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.927514 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.927922 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.927875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-client-certs\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.928253 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.928234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-tls\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.928673 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.928650 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-client-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.938115 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.938090 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpw5\" (UniqueName: \"kubernetes.io/projected/0328b8ce-272d-4232-8681-ef9e0d3a5274-kube-api-access-gcpw5\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:36.979585 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:36.979433 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:37.056722 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.056521 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r"] Oct 30 21:28:37.059937 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.059373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.068855 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.068832 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:28:37.068988 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.068974 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pgqtn\"" Oct 30 21:28:37.108179 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.108152 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r"] Oct 30 21:28:37.228169 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.228145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/129d99fe-11f5-478f-b470-0908f5a42037-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.228297 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.228273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwx4\" (UniqueName: \"kubernetes.io/projected/129d99fe-11f5-478f-b470-0908f5a42037-kube-api-access-hkwx4\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.266597 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.266571 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bdd5667d8-qdt5r"] Oct 30 21:28:37.276001 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:37.275970 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0328b8ce_272d_4232_8681_ef9e0d3a5274.slice/crio-0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec WatchSource:0}: Error finding container 0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec: Status 404 returned error can't find the container with id 0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec Oct 30 21:28:37.328742 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.328640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwx4\" (UniqueName: \"kubernetes.io/projected/129d99fe-11f5-478f-b470-0908f5a42037-kube-api-access-hkwx4\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.328742 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.328722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/129d99fe-11f5-478f-b470-0908f5a42037-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.331084 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.330274 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"024a6e401262fac20a691402a2fa042c16f2a2bf285f441548a469f626ed2840"} Oct 30 21:28:37.331084 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.330338 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"6d5e3639a9b0c4a9bc5a39c15dfb5e39f9304d56309d039a8419d33b5dd318dc"} Oct 30 21:28:37.331871 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.331744 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/129d99fe-11f5-478f-b470-0908f5a42037-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.334897 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.334877 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df"} Oct 30 21:28:37.335020 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.334925 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51"} Oct 30 21:28:37.335020 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.334978 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588"} Oct 30 21:28:37.335020 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.335017 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af"} Oct 30 21:28:37.335182 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.335047 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a"} Oct 30 21:28:37.336189 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.336170 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" event={"ID":"0328b8ce-272d-4232-8681-ef9e0d3a5274","Type":"ContainerStarted","Data":"0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec"} Oct 30 21:28:37.341059 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.341039 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwx4\" (UniqueName: \"kubernetes.io/projected/129d99fe-11f5-478f-b470-0908f5a42037-kube-api-access-hkwx4\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.386280 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.386258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:37.669636 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:37.669590 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r"] Oct 30 21:28:37.669992 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:37.669809 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129d99fe_11f5_478f_b470_0908f5a42037.slice/crio-4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0 WatchSource:0}: Error finding container 4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0: Status 404 returned error can't find the container with id 4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0 Oct 30 21:28:38.347225 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:38.347194 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"a4e958db3b7af6bfcbfd32c150201357ad16a485905f9495ebf566214900d203"} Oct 30 21:28:38.347321 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:38.347277 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"233d93efce9d3b7269d9db7fc1d68a0151e55129c80b56a84695a5eb5fc1cde1"} Oct 30 21:28:38.347359 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:38.347319 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"846ff9007b150feb12d2e4cb0ce7013212ebc97ff2b75dcc5a03b206f69175cc"} Oct 30 21:28:38.349450 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:38.349386 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" event={"ID":"129d99fe-11f5-478f-b470-0908f5a42037","Type":"ContainerStarted","Data":"4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0"} Oct 30 21:28:38.353785 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:38.353757 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerStarted","Data":"e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4"} Oct 30 21:28:38.398879 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:38.398836 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.8098593840000001 podStartE2EDuration="5.398823779s" totalImagesPullingTime="3.579162185s" totalInitContainerRuntime="9.80221ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:38.396339797 +0000 UTC m=+51.157666426" watchObservedRunningTime="2026-10-30 21:28:38.398823779 +0000 UTC m=+51.160150407" Oct 30 21:28:39.359012 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.358902 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" event={"ID":"0328b8ce-272d-4232-8681-ef9e0d3a5274","Type":"ContainerStarted","Data":"4f66af9386e53f90529a60f2f4e1054c204c2d6da2daa5222d0af45db0d7efca"} Oct 30 21:28:39.361851 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.361824 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" event={"ID":"b4e22dd8-f90b-4512-9ade-3f3ea3329946","Type":"ContainerStarted","Data":"7e7f52b4be4b3160dc17f050de50338c7ec4646b1b6262f91a6719c27ddfd4f0"} Oct 30 21:28:39.362023 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.361981 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:39.363450 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.363429 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ff4d698-mbksn" event={"ID":"3231e522-a00c-4ce7-9285-fe3954981036","Type":"ContainerStarted","Data":"b3a9b1381f88633d04a9b7801c64d9f97cdb3ad0079ef7b20bfef5ae3dad920b"} Oct 30 21:28:39.364903 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.364881 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" event={"ID":"129d99fe-11f5-478f-b470-0908f5a42037","Type":"ContainerStarted","Data":"a7cfbfc555d19b195e4ef93f45e56152292fa238af14a30cd4e427b080ffa368"} Oct 30 21:28:39.365237 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.365221 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:39.369486 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.369469 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:39.389498 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.389457 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" podStartSLOduration=1.68927311 podStartE2EDuration="3.389445511s" totalImagesPullingTime="1.700172401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:39.386860195 +0000 UTC m=+52.148186827" watchObservedRunningTime="2026-10-30 21:28:39.389445511 +0000 UTC m=+52.150772140" Oct 30 21:28:39.432779 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.432727 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" podStartSLOduration=2.381561418 podStartE2EDuration="5.432709372s" totalImagesPullingTime="3.051147954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:39.430615664 +0000 UTC m=+52.191942293" watchObservedRunningTime="2026-10-30 21:28:39.432709372 +0000 UTC m=+52.194036002" Oct 30 21:28:39.490424 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.490376 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-546ff4d698-mbksn" podStartSLOduration=1.9025775230000002 podStartE2EDuration="5.490363182s" totalImagesPullingTime="3.587785659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:39.490277109 +0000 UTC m=+52.251603738" watchObservedRunningTime="2026-10-30 21:28:39.490363182 +0000 UTC m=+52.251689811" Oct 30 21:28:39.519481 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:39.519438 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" podStartSLOduration=1.041337265 podStartE2EDuration="2.519427546s" totalImagesPullingTime="1.478090281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:39.518290802 +0000 UTC m=+52.279617428" watchObservedRunningTime="2026-10-30 21:28:39.519427546 +0000 UTC m=+52.280754174" Oct 30 21:28:41.863072 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:41.863038 2580 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546ff4d698-mbksn"] Oct 30 21:28:44.321526 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:44.321498 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:44.492260 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:44.492230 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:45.375318 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:45.375292 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:48.283978 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:48.283043 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:52.469763 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.469729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:52.469763 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.469767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:52.477277 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.477256 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:28:52.477365 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.477283 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:28:52.482520 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.482496 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:52.482605 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.482520 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:52.570878 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.570851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:52.573925 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.573909 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.585693 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.585676 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.594972 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.594939 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:52.621117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.621099 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjjj4\"" Oct 30 21:28:52.627520 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.627505 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4rkh\"" Oct 30 21:28:52.628571 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.628552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:52.633036 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.633009 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpdb2\"" Oct 30 21:28:52.635204 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.635193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:52.639985 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.639968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:52.921347 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.921320 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2nkx"] Oct 30 21:28:52.924498 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.924476 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sq9rx"] Oct 30 21:28:52.924582 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:52.924561 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75908611_03b2_42df_85e5_128bd14c5a43.slice/crio-3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024 WatchSource:0}: Error finding container 3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024: Status 404 returned error can't find the container with id 3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024 Oct 30 21:28:52.926673 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:52.926652 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c3c33e_a59d_4ae8_abec_366b2effbb07.slice/crio-ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f WatchSource:0}: Error finding container ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f: Status 404 returned error can't find the container with id ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f Oct 30 21:28:52.931735 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:52.931716 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mq6jf"] Oct 30 21:28:52.934081 ip-10-0-137-168 kubenswrapper[2580]: W1030 21:28:52.934060 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8be18f_c1ce_46ae_b36c_df83e438c101.slice/crio-f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317 WatchSource:0}: Error finding container f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317: Status 404 returned error can't find the container with id f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317 Oct 30 21:28:53.420159 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:53.420085 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sq9rx" event={"ID":"09c3c33e-a59d-4ae8-abec-366b2effbb07","Type":"ContainerStarted","Data":"ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f"} Oct 30 21:28:53.421407 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:53.421377 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mq6jf" event={"ID":"5f8be18f-c1ce-46ae-b36c-df83e438c101","Type":"ContainerStarted","Data":"f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317"} Oct 30 21:28:53.423206 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:53.423178 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2nkx" event={"ID":"75908611-03b2-42df-85e5-128bd14c5a43","Type":"ContainerStarted","Data":"3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024"} Oct 30 21:28:54.440635 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:54.440597 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2nkx" event={"ID":"75908611-03b2-42df-85e5-128bd14c5a43","Type":"ContainerStarted","Data":"2a35b202fd3bc9ecc847d8e4738f14b04d73b8af46d3a963ec454021bddbbd04"} Oct 30 21:28:54.441117 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:54.440665 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2nkx" event={"ID":"75908611-03b2-42df-85e5-128bd14c5a43","Type":"ContainerStarted","Data":"315ad130cd1beee35bdc5b0f658e282481c92e67d04ebbe22801d7b42ef1c137"} Oct 30 21:28:54.467297 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:54.466035 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p2nkx" podStartSLOduration=65.382683299 podStartE2EDuration="1m6.466020155s" totalImagesPullingTime="1.083336856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:54.465305766 +0000 UTC m=+67.226632396" watchObservedRunningTime="2026-10-30 21:28:54.466020155 +0000 UTC m=+67.227346789" Oct 30 21:28:56.450470 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:56.450378 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sq9rx" event={"ID":"09c3c33e-a59d-4ae8-abec-366b2effbb07","Type":"ContainerStarted","Data":"6e3e7d6b5687938bf23ee38aa15f290c322b1bbb943a8e9911a8dd145991639a"} Oct 30 21:28:56.450470 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:56.450469 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:56.470411 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:56.470353 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sq9rx" podStartSLOduration=65.234409352 podStartE2EDuration="1m8.470337403s" totalImagesPullingTime="3.235928051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:56.470119472 +0000 UTC m=+69.231446101" watchObservedRunningTime="2026-10-30 21:28:56.470337403 +0000 UTC m=+69.231664033" Oct 30 21:28:56.980130 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:56.980089 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:56.980328 ip-10-0-137-168 kubenswrapper[2580]: I1030 21:28:56.980159 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:57.739594 ip-10-0-137-168 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:28:57.761391 ip-10-0-137-168 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:28:57.761562 ip-10-0-137-168 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:28:57.761743 ip-10-0-137-168 systemd[1]: kubelet.service: Consumed 4.042s CPU time. Oct 30 21:28:57.773457 ip-10-0-137-168 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:28:57.836130 ip-10-0-137-168 kubenswrapper[6808]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:28:57.836130 ip-10-0-137-168 kubenswrapper[6808]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:28:57.836130 ip-10-0-137-168 kubenswrapper[6808]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:28:57.836130 ip-10-0-137-168 kubenswrapper[6808]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:28:57.838475 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838387 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.838475 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838471 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.838475 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838476 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.838475 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838480 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838485 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838489 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838493 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838497 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838501 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838505 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838509 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838513 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838517 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838521 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838525 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838529 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838533 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838537 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838541 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838544 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838549 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838552 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838556 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.838747 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838560 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838565 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838570 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838598 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838604 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838609 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838614 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838618 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838623 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838627 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838632 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838636 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838642 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838646 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838650 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838655 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838659 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838664 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838668 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.839634 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838673 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838677 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838681 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838687 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838691 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838695 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838700 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838704 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838708 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838713 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838717 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838721 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838726 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838730 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838734 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838738 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838751 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838756 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838761 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838765 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.840556 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838769 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838774 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838778 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838783 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838787 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838794 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838800 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838805 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838809 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838815 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838820 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838825 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838831 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838836 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838841 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838846 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838851 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838855 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838859 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838864 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.841440 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838868 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838873 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838878 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838882 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838887 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838891 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838895 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838900 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838903 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838908 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838912 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838917 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838921 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838926 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838930 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838934 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838938 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838942 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838965 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838972 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.842358 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838977 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838982 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838986 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.838991 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840216 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840227 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840232 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840236 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840241 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840247 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840251 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840258 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840264 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840270 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840276 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840281 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840285 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840290 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840295 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.843348 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840299 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840304 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840308 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840313 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840317 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840322 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840327 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840331 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840335 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840339 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840344 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840349 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840354 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840359 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840363 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840367 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840372 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840376 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840381 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840385 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.844261 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840389 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840395 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840402 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840413 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840418 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840422 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840428 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840432 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840436 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840440 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840445 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840449 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840453 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840458 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840462 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840467 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840472 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840476 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840481 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.845063 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840485 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840491 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840495 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840500 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840504 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840508 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840512 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840516 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840521 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840525 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840530 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840534 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840538 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840542 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840546 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840551 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840557 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840562 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840566 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840570 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.845555 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840575 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840579 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840583 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840588 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840593 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840597 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840602 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840607 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840611 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840615 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840620 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840624 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840628 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840632 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840637 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840641 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840645 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840650 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840654 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840658 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.846080 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840663 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840669 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840673 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840678 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840682 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840686 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840691 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840695 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840700 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840704 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840709 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.840713 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841596 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841607 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841612 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841617 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841621 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841626 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841630 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841634 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.846630 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841639 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841643 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841647 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841652 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841656 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841660 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841665 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841669 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841673 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841677 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841682 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841688 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841693 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841697 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841701 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841705 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841717 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841721 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841725 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841730 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.847279 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841735 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841740 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841745 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841750 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841754 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841758 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841763 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841767 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841771 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841776 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841780 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841785 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841789 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841794 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841798 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841802 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841806 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841810 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841815 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841819 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.847836 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841823 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841828 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841832 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841837 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841841 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841845 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841850 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841854 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841858 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841862 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841866 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841870 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841875 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841880 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841884 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841889 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841893 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841897 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841905 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.848441 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841911 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841915 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841920 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841925 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841929 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841936 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841941 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841961 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841965 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841969 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841974 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841978 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841982 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841987 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841991 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.841995 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842000 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842004 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842008 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.848934 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842012 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842016 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842021 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842025 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842029 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842033 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842038 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842042 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842046 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842051 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842056 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842060 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842064 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842068 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842073 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842077 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842085 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842091 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842100 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.842104 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.849418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842239 6808 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842266 6808 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842277 6808 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842287 6808 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842294 6808 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842300 6808 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842306 6808 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842313 6808 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842318 6808 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842323 6808 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842329 6808 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842334 6808 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842339 6808 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842344 6808 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842349 6808 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842353 6808 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842358 6808 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842362 6808 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842369 6808 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842373 6808 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842378 6808 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842383 6808 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842388 6808 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842396 6808 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:28:57.849936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842401 6808 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842407 6808 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842412 6808 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842416 6808 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842422 6808 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842426 6808 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842434 6808 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842439 6808 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842446 6808 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842451 6808 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842456 6808 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842463 6808 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842467 6808 flags.go:64] FLAG: --enable-server="true" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842472 6808 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842480 6808 flags.go:64] FLAG: --event-burst="100" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842485 6808 flags.go:64] FLAG: --event-qps="50" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842490 6808 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842495 6808 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842500 6808 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842506 6808 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842511 6808 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842516 6808 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842521 6808 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842526 6808 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842531 6808 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:28:57.850525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842536 6808 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842541 6808 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842546 6808 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842551 6808 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842555 6808 flags.go:64] FLAG: --feature-gates="" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842562 6808 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842567 6808 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842573 6808 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842579 6808 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842583 6808 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842588 6808 flags.go:64] FLAG: --help="false" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842593 6808 flags.go:64] FLAG: --hostname-override="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842598 6808 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842605 6808 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842610 6808 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842616 6808 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842622 6808 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842626 6808 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842633 6808 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842638 6808 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842642 6808 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842647 6808 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842652 6808 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842657 6808 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:28:57.851360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842663 6808 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842668 6808 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842673 6808 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842677 6808 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842682 6808 flags.go:64] FLAG: --lock-file="" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842687 6808 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842691 6808 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842696 6808 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842712 6808 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842716 6808 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842721 6808 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842726 6808 flags.go:64] FLAG: --logging-format="text" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842731 6808 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842737 6808 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842741 6808 flags.go:64] FLAG: --manifest-url="" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842746 6808 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842754 6808 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842759 6808 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842766 6808 flags.go:64] FLAG: --max-pods="110" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842771 6808 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842776 6808 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842782 6808 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842787 6808 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842792 6808 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842796 6808 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:28:57.852123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842801 6808 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842814 6808 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842819 6808 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842824 6808 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842829 6808 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842833 6808 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842838 6808 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842843 6808 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842848 6808 flags.go:64] FLAG: --port="10250" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842853 6808 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842857 6808 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ec740e4f156883e0" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842863 6808 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842867 6808 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842872 6808 flags.go:64] FLAG: --register-node="true" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842877 6808 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842883 6808 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842888 6808 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842892 6808 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842897 6808 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842903 6808 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842908 6808 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842913 6808 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842919 6808 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842924 6808 flags.go:64] FLAG: --runonce="false" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842931 6808 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842937 6808 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842942 6808 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:28:57.852849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842964 6808 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842971 6808 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842976 6808 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842981 6808 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842986 6808 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842990 6808 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.842996 6808 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843001 6808 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843006 6808 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843011 6808 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843016 6808 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843021 6808 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843030 6808 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843034 6808 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843039 6808 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843046 6808 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843051 6808 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843055 6808 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843061 6808 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843066 6808 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843071 6808 flags.go:64] FLAG: --v="2" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843077 6808 flags.go:64] FLAG: --version="false" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843084 6808 flags.go:64] FLAG: --vmodule="" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843090 6808 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843096 6808 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:28:57.854010 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843309 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843318 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843324 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843329 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843333 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843339 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843343 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843347 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843353 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843357 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843361 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843365 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843369 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843376 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843381 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843385 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843390 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843394 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843398 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843404 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.855105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843408 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843413 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843417 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843421 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843425 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843429 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843434 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843438 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843442 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843447 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843451 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843455 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843461 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843467 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843472 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843478 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843482 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843487 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843492 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.855746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843496 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843503 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843507 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843512 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843516 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843521 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843527 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843532 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843537 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843541 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843546 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843550 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843554 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843559 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843563 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843568 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843573 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843578 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843582 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.856367 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843586 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843591 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843595 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843599 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843603 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843607 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843612 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843616 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843621 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843625 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843629 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843634 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843638 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843649 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843655 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843660 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843664 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843669 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843673 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843679 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.856894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843684 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843688 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843693 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843700 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843706 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843712 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843717 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843721 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843725 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843729 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843734 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843738 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843742 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843746 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843750 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843755 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843759 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843764 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843768 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.857523 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843772 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843777 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843781 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843785 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843789 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843793 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843798 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843805 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843810 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.843852 6808 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843919 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843927 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843932 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843939 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843943 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843964 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.858055 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843969 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843974 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843978 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843982 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843986 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843991 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843995 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.843999 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844003 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844008 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844012 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844017 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844021 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844026 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844030 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844035 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844039 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844043 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844047 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.858485 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844051 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844056 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844060 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844064 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844071 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844080 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844085 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844091 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844098 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844103 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844110 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844115 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844119 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844123 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844127 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844132 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844136 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844140 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844145 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.859051 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844149 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844153 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844158 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844162 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844166 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844171 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844175 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844179 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844184 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844188 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844193 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844200 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844204 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844208 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844212 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844216 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844221 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844227 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844231 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.859662 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844237 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844241 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844245 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844250 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844256 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844260 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844264 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844268 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844272 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844276 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844281 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844285 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844290 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844294 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844299 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844304 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844308 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844312 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844319 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.860259 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844326 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844333 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844339 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844345 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844351 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844355 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844360 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844364 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844369 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844373 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844377 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844384 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844389 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844394 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844399 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844403 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844408 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844413 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844417 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844422 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.860763 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844426 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844431 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844435 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.844440 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.848616 6808 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.848633 6808 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.848660 6808 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.848666 6808 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848724 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848729 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848733 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848736 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848738 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848741 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848744 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848747 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848750 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848753 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848756 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848759 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.861331 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848761 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848764 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848767 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848770 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848773 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848776 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848778 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848781 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848785 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848789 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848792 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848795 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848798 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848801 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848804 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848807 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848809 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848818 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848822 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.861834 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848825 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848827 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848830 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848832 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848835 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848838 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848841 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848844 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848846 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848849 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848852 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848854 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848857 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848859 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848862 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848864 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848868 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848871 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848873 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.862316 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848876 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848878 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848881 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848883 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848886 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848889 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848891 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848894 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848896 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848899 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848901 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848904 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848912 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848916 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848918 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848921 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848924 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848928 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848932 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.862821 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848935 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848938 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848941 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848943 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848961 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848965 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848967 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848970 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848973 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848976 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848979 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848982 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848984 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848987 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848989 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848992 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848995 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.848998 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849000 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849003 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.863428 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849006 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849008 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849011 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849014 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849016 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849019 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849028 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849031 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849033 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849036 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849038 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849041 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849044 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849046 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849049 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849051 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.864121 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849054 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.849081 6808 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849263 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849269 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849272 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849275 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849278 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849280 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849283 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849286 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849288 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849291 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849294 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849296 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849299 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849301 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.864720 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849304 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849307 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849309 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849312 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849314 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849317 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849320 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849328 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849331 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849333 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849336 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849338 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849341 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849344 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849346 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849349 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849351 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849354 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849356 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.865378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849364 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849367 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849370 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849372 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849375 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849378 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849380 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849382 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849385 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849388 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849390 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849393 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849396 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849399 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849402 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849405 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849407 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849410 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849412 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.865881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849415 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849418 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849428 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849431 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849434 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849436 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849439 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849441 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849444 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849446 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849449 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849451 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849454 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849457 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849459 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849462 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849464 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849467 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849469 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849472 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.866378 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849474 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849477 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849479 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849482 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849485 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849487 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849490 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849492 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849495 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849498 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849500 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849503 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849507 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849511 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849514 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849522 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849525 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849528 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849531 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.866879 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849533 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849536 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849538 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849541 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849543 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849546 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849549 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849551 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849554 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849556 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849559 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849561 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849564 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849566 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849569 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.849592 6808 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:28:57.867421 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849629 6808 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849633 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849636 6808 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849639 6808 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849642 6808 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849644 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849647 6808 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849649 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849652 6808 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849654 6808 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849657 6808 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849659 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849662 6808 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849670 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849673 6808 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849675 6808 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849678 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849680 6808 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849683 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849685 6808 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:57.867802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849688 6808 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849690 6808 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849693 6808 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849696 6808 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849699 6808 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849701 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849704 6808 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849707 6808 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849716 6808 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849719 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849722 6808 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849724 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849727 6808 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849730 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849732 6808 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849735 6808 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849738 6808 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849740 6808 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849742 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849745 6808 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:57.868288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849747 6808 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849750 6808 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849752 6808 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849755 6808 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849758 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849760 6808 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849770 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849773 6808 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849776 6808 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849778 6808 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849781 6808 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849783 6808 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849786 6808 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849788 6808 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849791 6808 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849793 6808 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849796 6808 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849799 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849802 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:57.869168 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849805 6808 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849807 6808 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849810 6808 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849813 6808 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849815 6808 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849820 6808 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849824 6808 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849827 6808 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849830 6808 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849833 6808 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849835 6808 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849838 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849841 6808 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849843 6808 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849846 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849849 6808 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849851 6808 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849854 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849857 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:57.869855 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849859 6808 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849867 6808 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849870 6808 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849873 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849875 6808 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849878 6808 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849880 6808 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849883 6808 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849886 6808 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849888 6808 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849891 6808 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849893 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849896 6808 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849898 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849901 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849903 6808 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849906 6808 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849908 6808 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849915 6808 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849919 6808 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:57.870394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849922 6808 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849925 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849928 6808 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849930 6808 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849934 6808 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849937 6808 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849939 6808 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:28:57.849942 6808 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.850014 6808 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851031 6808 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851079 6808 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851282 6808 server.go:1014] "Starting client certificate rotation" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851419 6808 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851472 6808 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:22:48 +0000 UTC" deadline="2028-04-03 12:30:15.447330647 +0000 UTC" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851539 6808 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12495h1m17.595794762s" Oct 30 21:28:57.870911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.851890 6808 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:28:57.871406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.854929 6808 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:28:57.871406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.856439 6808 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:28:57.871406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.859069 6808 log.go:25] "Validated CRI v1 image API" Oct 30 21:28:57.871406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.859900 6808 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:28:57.871406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.862171 6808 fs.go:135] Filesystem UUIDs: map[2efac825-e414-46f9-a17e-a23aac127836:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b2b6838c-4110-468b-80c0-c491f8f690a9:/dev/nvme0n1p3] Oct 30 21:28:57.871566 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.862192 6808 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ee47fc53bc4db11db3c00a0bb3be2f0658cb61a4aa06cbbb5e1e3c7ce9b7182/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ee47fc53bc4db11db3c00a0bb3be2f0658cb61a4aa06cbbb5e1e3c7ce9b7182/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32a94c505a332ec8116af313fa49d2586dc06556412c624074ee777aa7f279be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32a94c505a332ec8116af313fa49d2586dc06556412c624074ee777aa7f279be/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43d97fae832e343d1246c7ac616ce94a2c953740df2c4936bd32c18716121a5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43d97fae832e343d1246c7ac616ce94a2c953740df2c4936bd32c18716121a5d/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0/userdata/shm major:0 minor:394 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f/userdata/shm major:0 minor:444 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7761f3f0956a26e38bf54f450c1cb28638e0633a2a158ff7f3892f146942b13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7761f3f0956a26e38bf54f450c1cb28638e0633a2a158ff7f3892f146942b13/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e/userdata/shm major:0 minor:53 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e/userdata/shm major:0 minor:54 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317/userdata/shm major:0 minor:446 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~projected/kube-api-access-gcpw5:{mountpoint:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~projected/kube-api-access-gcpw5 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09c3c33e-a59d-4ae8-abec-366b2effbb07/volumes/kubernetes.io~projected/kube-api-access-d5j5r:{mountpoint:/var/lib/kubelet/pods/09c3c33e-a59d-4ae8-abec-366b2effbb07/volumes/kubernetes.io~projected/kube-api-access-d5j5r major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/129d99fe-11f5-478f-b470-0908f5a42037/volumes/kubernetes.io~projected/kube-api-access-hkwx4:{mountpoint:/var/lib/kubelet/pods/129d99fe-11f5-478f-b470-0908f5a42037/volumes/kubernetes.io~projected/kube-api-access-hkwx4 major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/129d99fe-11f5-478f-b470-0908f5a42037/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/129d99fe-11f5-478f-b470-0908f5a42037/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~projected/kube-api-access-wckqp:{mountpoint:/var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~projected/kube-api-access-wckqp major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~projected/kube-api-access-f28vh:{mountpoint:/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~projected/kube-api-access-f28vh major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada/volumes/kubernetes.io~projected/kube-api-access-6ddgd:{mountpoint:/var/lib/kubelet/pods/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada/volumes/kubernetes.io~projected/kube-api-access-6ddgd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~projected/kube-api-access-xxz9l:{mountpoint:/var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~projected/kube-api-access-xxz9l major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4422cee8-aec1-4569-9277-bba73b00a9ad/volumes/kubernetes.io~projected/kube-api-access-hclrn:{mountpoint:/var/lib/kubelet/pods/4422cee8-aec1-4569-9277-bba73b00a9ad/volumes/kubernetes.io~projected/kube-api-access-hclrn major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55b6c776-a4e5-492f-a017-3c9dfc567d56/volumes/kubernetes.io~projected/kube-api-access-x5wgs:{mountpoint:/var/lib/kubelet/pods/55b6c776-a4e5-492f-a017-3c9dfc567d56/volumes/kubernetes.io~projected/kube-api-access-x5wgs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd38803-1d13-470a-97b1-99bc9262d8f0/volumes/kubernetes.io~projected/kube-api-access-jxshq:{mountpoint:/var/lib/kubelet/pods/5dd38803-1d13-470a-97b1-99bc9262d8f0/volumes/kubernetes.io~projected/kube-api-access-jxshq major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f8be18f-c1ce-46ae-b36c-df83e438c101/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5f8be18f-c1ce-46ae-b36c-df83e438c101/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75908611-03b2-42df-85e5-128bd14c5a43/volumes/kubernetes.io~projected/kube-api-access-ng5k6:{mountpoint:/var/lib/kubelet/pods/75908611-03b2-42df-85e5-128bd14c5a43/volumes/kubernetes.io~projected/kube-api-access-ng5k6 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75908611-03b2-42df-85e5-128bd14c5a43/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/75908611-03b2-42df-85e5-128bd14c5a43/volumes/kubernetes.io~secret/metrics-certs major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89806e93-18c5-43a5-b1ed-0f6eae4f20f3/volumes/kubernetes.io~projected/kube-api-access-lktlq:{mountpoint:/var/lib/kubelet/pods/89806e93-18c5-43a5-b1ed-0f6eae4f20f3/volumes/kubernetes.io~projected/kube-api-access-lktlq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d852c03-e50d-4039-bd52-7bfae081a271/volumes/kubernetes.io~projected/kube-api-access-r8jcz:{mountpoint:/var/lib/kubelet/pods/8d852c03-e50d-4039-bd52-7bfae081a271/volumes/kubernetes.io~projected/kube-api-access-r8jcz major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d852c03-e50d-4039-bd52-7bfae081a271/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8d852c03-e50d-4039-bd52-7bfae081a271/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9233cc4d-86f9-4890-9464-33c7eefd42a8/volumes/kubernetes.io~projected/kube-api-access-ds64h:{mountpoint:/var/lib/kubelet/pods/9233cc4d-86f9-4890-9464-33c7eefd42a8/volumes/kubernetes.io~projected/kube-api-access-ds64h major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e6b899c-d914-4adb-9a7c-230e359b1497/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9e6b899c-d914-4adb-9a7c-230e359b1497/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9f17f5b-18dc-4642-a2bb-d778b0524561/volumes/kubernetes.io~projected/kube-api-access-trx6c:{mountpoint:/var/lib/kubelet/pods/a9f17f5b-18dc-4642-a2bb-d778b0524561/volumes/kubernetes.io~projected/kube-api-access-trx6c major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9f17f5b-18dc-4642-a2bb-d778b0524561/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a9f17f5b-18dc-4642-a2bb-d778b0524561/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volumes/kubernetes.io~projected/kube-api-access-sbqwg:{mountpoint:/var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volumes/kubernetes.io~projected/kube-api-access-sbqwg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~projected/kube-api-access-gck7r:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~projected/kube-api-access-gck7r major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/alertmanager/9 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/config-reloader/6 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~projected/kube-api-access-f6v7s:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~projected/kube-api-access-f6v7s major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~projected/tls-assets major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/config-volume major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/web-config major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4693396-79bd-410d-86a3-5a61f614d231/volumes/kubernetes.io~projected/kube-api-access-88c2z:{mountpoint:/var/lib/kubelet/pods/c4693396-79bd-410d-86a3-5a61f614d231/volumes/kubernetes.io~projected/kube-api-access-88c2z major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4693396-79bd-410d-86a3-5a61f614d231/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c4693396-79bd-410d-86a3-5a61f614d231/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~projected/kube-api-access-82bsw:{mountpoint:/var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~projected/kube-api-access-82bsw major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~projected/kube-api-access-v254t:{mountpoint:/var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~projected/kube-api-access-v254t major:0 minor:50 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ab67995a8c63b0a0bce504d7c9b6be29792994c827b1f8f2270e2538cb89fc2e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ee863bf71a81d69def796ece95f1c7ba2b57e65a648f05c6f4ea6b6a8b6c8ae7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/aee08ced398d87e156805088b603bfcf58b4cb360a1164f67cb20446f113d897/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7fa874fb986eab4a811561e82b11ac8f5e71ca8885d93d33c1bc08e80b3e5c9c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/39a1eca1d53c03f9e17efb9d1b40a2f12357e5a3a1c0963fed0a4276ebcf6e89/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5b5e9a847f9e3d37fb925e311663c35f82dfcde30fa9421304a9dee9236a3b95/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/21c12628e5e36ab65652ac06feb254f83065345c262fad84adf342cd9254258a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e98d652957532822142615869c168c6becdc52885fd1fe526ce52afa827c108f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4b5490440b9853b51dcbd1fe87b51f49151ecb788c7d3b5491524fcc79a659c8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a000a0dab252ba139df46702414f5d1e02620a0a65abe37f03ded672d41bb36d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/adab34a5d8bd09e299728f467b352e85e91512c838cadf817a96f313415c2913/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/22c06abcee647d8ac4da6ea0d0d27ea3d0652dcb544c32e6cc21fb1ef7cc8117/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6c3d055b661bc48a33fd4ab127c62d509f68795cd1f679ff35648c3deca2a8dc/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/33adb18c99e06061fe83b154e44bd5b288d55c6279066d3cd30b8e5a0f58e3fb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dc7f6fcdb37bc3916a6924007db0091d669d613cdc51a3bb7ef49ba81d3c40bc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bd27f6cc7a070142b1cfeab5626267a3c04b1e071df22a5f3568fd2a000fa057/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/332f6a3636a53c45fcd6021b5c5850e8a956dfadbdada98bd251a7fcd34df688/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7e04ea3096518d3b229489199490c2f9faaaecebaa5b28a419fafabb9d5a4c81/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c95b7fe3d137bc8da4c44c517884f9d9ef863c2360bc868b1c481151850fe35f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d3cdba19aedc3ea7814b1df5553dbb6979f886b993746715e97364eaced0f1c9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1cd968a228af44ffca7b7bcd95b67322b0806afc8d230b0bc7c375e1b77be0ef/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/77e5b38d1c2f441f1d28cadbdc7453153cf2bb7b39a4e15199fb582f10fdda8e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/eb32d68b70018beab9707c68fedfe00b07163daf32a7384112532988a3baf5ac/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/31476cdc4541bfd74b9502cfd055279f1b34bd325f717382a7e0390ad316c323/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2ef7457f4f34d5855c268f5ca37de722799f371f8ea9a319c9e170148e30ab69/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/c919af4b1970245674a6f5a899c117ee7599d025eabc38c17bdad4179a0a11b4/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/cd624c5e11777fd96ac12e817974bcc20593aa361d9039840a34c15e41a96981/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/cbff9ca4540c61ff1a60e028b83b6495e48012e5f8f5c1320615695abaf2688e/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/892fad7c5c02b73fff1bba12579c2fccb0c82b395215431871a09ba8e9ceb7f2/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/e8f4694d9b3e8ce0e0eaddc7f5b5a752247a076334f423cc00e0ddca7ac981fd/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/68d5577c251f64ef1c09bb05aa2e018cb6157f451704532603d1e78210a21c66/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/a8abd1702cd5953cd520b005667f7cadabb9674ad87d6e1c8e27b350bb029ff3/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/608764d66fe94bfe871f7144e47d8c15394e42a27a83feec28596931c7dec924/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/85c2ebf868957fc7bfa633375b8c3aaf83563673b3cc90d45d7722ae8f91eabc/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/ae03d1356d600441a5ed6bb0223916327d6325ef00093612678e5ca587578c74/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/27437cd47d4e1470d42a0fc73bbde07a515109dcadd70f107cb2e42e708b7eb8/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/c63f766ecce0814d2ace6bdf7508ecbe12726c004d05ec6ae113a49617aad9f0/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/70c68999c04e3008c1a4d422b2e86427d14e429b23a04805b6fde67f8ddb6a63/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/71d7a1bef7a998cd430e06c1b530b5ecd7accd9e9794fb4057d7362e08a6ba23/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/cc1ccb14c1b4ccba5f7fc682d1ad25bd9dc402a5340ce86ba7841257666e52df/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/31cb7e78b200f9fc1f5af2d1551893913091b961014dd0abe9c94bae2e3e5ce5/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/169a318960d9aa4a31642306afb0f45186975571dfefe5d2de870e195b98f376/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/b109fb0b8688136f000d5d22cc42d033c77ddd25be43f09f03c0eb9576b10556/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/f6851eea9fc3a149b892ec23ef08d458d9cb820259bc17653c32fbfed953ed7f/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/ad8b9c30511700e9ddf136ac422ea809c405988a5a57b4454aec7f18450ef983/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/ef40b1fa318114c1383d330da300b458b7632ffceb6ceb4af096f172501d65e4/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/8924f5a14541fc2d07961f080f8bcd518c3b53f6bc6c4d036c873534d311e926/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d2b905d05c7c4ef2562eec1eca01d6d8b47014fe1790b7c00f57fb0fd49f46de/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/21b875b5de93b420acb59b67e6915e755a1645672f8e02b2a2c435fa2eacc3d7/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/fc55ff4c267db64097cb2809145ec5fc3425df04e1629dc946bd3f01bb3387bf/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/a44b0ef5503ade6ce006371d309f3aefe3e3c78ca8ec7226e03eaacb60c98410/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/8e2e930369eacbeb1916ea7cc8747bd80ce1dbfeebdd6652ec8dece198102a5c/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/5684742c375c708f92490e29e23156a846222a0370f0290fe16859711a4d45f2/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/86ad6572b0e702dde4b0afb5c661010858effb35528a9cf1d4594927f98e71fb/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/cf30de4012ed1debde75af397df10b243cdb0563e1fb26b38857ed2a40038f92/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/c57ab06f9c6b3e12ac65f1305c65dd067e7bc68774d0dfc92504b010538d44ea/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/0ec0b1c49e78da959bad1d12f36db3800c943f03dcddfe5cf1f9cdd0cacc8b8b/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/3975a3ed7a3f28166abdd1d0ab4ef51abe59372ec711d6519d8dae581c25f112/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/3ac85431d0e0a4e401e455c23fc5b845315f413ee92647e7530550cd7244e949/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/db53162c8fb6ca265bde9769ded46da3ec858257cb5469f51df4139d46cb0998/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/2263a08cb46a381af230600c8c67b93f7b24741164655ddec256815c5b6cf9e3/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/5d1e0fd86e17e46c97523244b6193140dc607fc1a4b312475101e427589be1e6/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/5cf6d3c5701e2f01b182dcf183b53b5e0182eec3405e29047d50863b5ae189b0/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/c1434cabeec06d1015fcca8024768216a58536268c73b78dbbe1a0fcc021770a/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/081d437e5d2f29a34457f438c9936ba8b871a4fbec8f90ed687207cbab0e76bc/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/3b2ff177d24c358cda0b0b2b7b99c29a7d3254f450aeaa3270a506190fe091ff/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/e3a409a4372ae918e4d813e96b2c231aa9c2097a58a638b74d977b817f382ae1/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/acd60f3f830e02235d398d280611705558960f94b0ee2982b66c3a3c5c409112/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ce3898e6054ffab915350ebd045972bc0409d7fb975097fe318dd2d331024c02/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/e22dc27331319720bd25a688edeccf2df9c7aea58c57a0f0089e0fc3991c6ef6/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b67b2fbc4e35a456b39aef7dbc0cc04253a505fc2b701766a12551bde05b3478/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/7a18b7d6b0bee0ab58e2d1d7581b0e2c1250bf87ce1df5536f6d40f45ad4f3ab/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/21f3648e508645c91a2b5896d8ab56e85454d763553d113f73056ed04566a6a1/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/44ff2d2a0ffe7e8ac17504fea31ec7f1f813aa917801b353a6585c45ad1a47fb/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c8703723472cebe61f5b1ca370632c8a0d8e3d848d1b2c4a535b3f15eefb964e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/668f10c89655de107505a9176ab6d93eaa536626ed74da8050aa6dfe860f3e7d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d5907d8672113f04a492d621b16164fa7fa4464d2d04784111f17642b9bbb41c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0dda6945cfaf1e8d5e156b4bcb823087dec5f3b3aa97edd2d37c1b3dec93a91b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.870104 6808 manager.go:217] Machine: {Timestamp:2026-10-30 21:28:57.86955582 +0000 UTC m=+0.065246937 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101847 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22783dd70c706f19907f7e05892a9e SystemUUID:ec22783d-d70c-706f-1990-7f7e05892a9e BootID:419bfb6a-37b0-4924-9149-12f297e0be26 Filesystems:[{Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada/volumes/kubernetes.io~projected/kube-api-access-6ddgd DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d852c03-e50d-4039-bd52-7bfae081a271/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09c3c33e-a59d-4ae8-abec-366b2effbb07/volumes/kubernetes.io~projected/kube-api-access-d5j5r DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~projected/kube-api-access-f28vh DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9233cc4d-86f9-4890-9464-33c7eefd42a8/volumes/kubernetes.io~projected/kube-api-access-ds64h DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c58622cf3be20632190de67a6625a069dfb4868a3f8428865517d10d8852041/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5051666ec24cec07b470d2643fc0deacac85433171ea6e85d0aed44a3193317/userdata/shm DeviceMajor:0 DeviceMinor:446 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75908611-03b2-42df-85e5-128bd14c5a43/volumes/kubernetes.io~projected/kube-api-access-ng5k6 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4401dd436562ed6cad032d47ab7c9a6d89c327b84330c92cafc37e9856394658/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4693396-79bd-410d-86a3-5a61f614d231/volumes/kubernetes.io~projected/kube-api-access-88c2z DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~projected/kube-api-access-f6v7s DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32a94c505a332ec8116af313fa49d2586dc06556412c624074ee777aa7f279be/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43d97fae832e343d1246c7ac616ce94a2c953740df2c4936bd32c18716121a5d/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd38803-1d13-470a-97b1-99bc9262d8f0/volumes/kubernetes.io~projected/kube-api-access-jxshq DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9f17f5b-18dc-4642-a2bb-d778b0524561/volumes/kubernetes.io~projected/kube-api-access-trx6c DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:268 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7761f3f0956a26e38bf54f450c1cb28638e0633a2a158ff7f3892f146942b13/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bea6b94d447da9d2b036fd7c6ee203333189cdf4dec6196a57c4ce891e07b0b/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bc83e51cacc7de4e61dc429a2808e94fac6e17260d4c0ae99d756769979bb7c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/129d99fe-11f5-478f-b470-0908f5a42037/volumes/kubernetes.io~projected/kube-api-access-hkwx4 DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e780051d3cb30d38a224693a9f38bb0a89724aa1a29d03408e49249fb5c84c0/userdata/shm DeviceMajor:0 DeviceMinor:394 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~projected/kube-api-access-v254t DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2aeae646ac1ce6fdd052b7eaee6dd0df071265a3902cd35b6167d2bc108614e/userdata/shm DeviceMajor:0 DeviceMinor:54 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3621674c47660df20909d71edcae66e008943b65f5a027183d3acf590b3ad07/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:268 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:268 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e6b899c-d914-4adb-9a7c-230e359b1497/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d852c03-e50d-4039-bd52-7bfae081a271/volumes/kubernetes.io~projected/kube-api-access-r8jcz DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~projected/kube-api-access-gck7r DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0a5185b581c29eb33c45048430e251df141515991a82059885581ac1f287c19/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~projected/kube-api-access-gcpw5 DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fe8216738f05d1bb6c7f5f3f2a365fda4b4501fc40ba7bbb6399bfade5190fa/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a7ac637686390ce8d8766fb3f076c4ad988a5b249ddeb3a66c2f88b93288644/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce0c798a8bf4c2297f3fe1948579d2f62462224b797d161b85788b0e05d3d18f/userdata/shm DeviceMajor:0 DeviceMinor:444 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ee47fc53bc4db11db3c00a0bb3be2f0658cb61a4aa06cbbb5e1e3c7ce9b7182/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e12e3e7f9c9bacbabeb546984bbd37c54931a1943afeb598f36483eb9c327ecf/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55b6c776-a4e5-492f-a017-3c9dfc567d56/volumes/kubernetes.io~projected/kube-api-access-x5wgs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf66f3413894cfd55c5efb940c5a8b419955452c643ac7ebf8d9ed2d22f07f54/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4422cee8-aec1-4569-9277-bba73b00a9ad/volumes/kubernetes.io~projected/kube-api-access-hclrn DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~projected/kube-api-access-xxz9l DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bcddb42f2337613ebba1d09acada2fd976ea6fb1c59252687b27097f33abfec/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f1eb5cce6f37a89d7112a45f85480ca058b70a729561287df60cf6fa1d99024/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f/volumes/kubernetes.io~projected/kube-api-access-sbqwg DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4693396-79bd-410d-86a3-5a61f614d231/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75908611-03b2-42df-85e5-128bd14c5a43/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0328b8ce-272d-4232-8681-ef9e0d3a5274/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89806e93-18c5-43a5-b1ed-0f6eae4f20f3/volumes/kubernetes.io~projected/kube-api-access-lktlq DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9f17f5b-18dc-4642-a2bb-d778b0524561/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65adbcb22ac3f01d6bbfae5aa29dfa16dce87775dda4496668bbc9e0bbbed0b1/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/129d99fe-11f5-478f-b470-0908f5a42037/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/volumes/kubernetes.io~projected/kube-api-access-82bsw DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~projected/kube-api-access-wckqp DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e22dd8-f90b-4512-9ade-3f3ea3329946/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e/userdata/shm DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/136a5087-3017-46af-aa81-15b1a899031a/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:268 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f8be18f-c1ce-46ae-b36c-df83e438c101/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:440 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0bcddb42f233761 MacAddress:c6:06:c2:b1:a1:80 Speed:10000 Mtu:8901} {Name:32a94c505a332ec MacAddress:c6:d5:23:49:26:ae Speed:10000 Mtu:8901} {Name:3f1eb5cce6f37a8 MacAddress:12:ba:95:99:0d:c3 Speed:10000 Mtu:8901} {Name:43d97fae832e343 MacAddress:fa:e2:0b:d5:5d:ca Speed:10000 Mtu:8901} {Name:4bc83e51cacc7de MacAddress:82:05:5b:be:df:65 Speed:10000 Mtu:8901} {Name:4e780051d3cb30d MacAddress:ce:dc:2d:57:33:01 Speed:10000 Mtu:8901} {Name:65adbcb22ac3f01 MacAddress:3e:f3:08:71:00:66 Speed:10000 Mtu:8901} {Name:75105fd7ab2e3ae MacAddress:7e:0d:c5:8f:f7:e6 Speed:10000 Mtu:8901} {Name:9bea6b94d447da9 MacAddress:fa:46:b7:1d:ab:be Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cd:c0:9e:07 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:49:2d:62:12:86 Speed:0 Mtu:8901} {Name:ce0c798a8bf4c22 MacAddress:ce:35:f4:cf:d5:8e Speed:10000 Mtu:8901} {Name:d0a5185b581c29e MacAddress:16:f5:be:3d:23:f6 Speed:10000 Mtu:8901} {Name:e12e3e7f9c9bacb MacAddress:66:88:78:c6:30:8f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cd:c0:9e:07 Speed:0 Mtu:9001} {Name:f5051666ec24cec MacAddress:2e:3f:a2:38:85:c3 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:04:38:86:3a:59 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:09:33:8d:f0:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.870515 6808 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.870559 6808 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.870995 6808 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871019 6808 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-168.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871224 6808 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871236 6808 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871250 6808 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871281 6808 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871380 6808 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871610 6808 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871737 6808 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871771 6808 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871781 6808 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871790 6808 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871819 6808 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:28:57.871867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.871827 6808 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:28:57.872604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.872537 6808 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:28:57.872604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.872547 6808 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:28:57.874604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.874383 6808 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:28:57.877325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877309 6808 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:28:57.877325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877323 6808 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:28:57.877443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877433 6808 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:28:57.877628 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877616 6808 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:28:57.877658 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877646 6808 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:28:57.877744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877733 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877747 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877754 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877760 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877766 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877771 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877777 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:28:57.877780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877783 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:28:57.878003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877789 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:28:57.878003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877795 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:28:57.878003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877804 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:28:57.878003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877813 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:28:57.878003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877836 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:28:57.878003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.877841 6808 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:28:57.881255 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.881237 6808 server.go:1263] "Started kubelet" Oct 30 21:28:57.881349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.881304 6808 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:28:57.882864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.881612 6808 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:28:57.882864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.881688 6808 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:28:57.882864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.881998 6808 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:28:57.881775 ip-10-0-137-168 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:28:57.888121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.887053 6808 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:28:57.888121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.888036 6808 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:28:57.888121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.888049 6808 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:28:57.888752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.888149 6808 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:28:57.888836 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.888786 6808 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:22:49 +0000 UTC" deadline="2028-04-03 07:26:47.437395638 +0000 UTC" Oct 30 21:28:57.888836 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.888821 6808 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12489h57m49.548578088s" Oct 30 21:28:57.888935 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:57.888867 6808 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-168.us-east-2.compute.internal\" not found" Oct 30 21:28:57.889939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.889918 6808 factory.go:55] Registering systemd factory Oct 30 21:28:57.889939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.889939 6808 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:28:57.891628 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.891607 6808 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:28:57.891880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.891855 6808 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:28:57.893343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.893323 6808 factory.go:153] Registering CRI-O factory Oct 30 21:28:57.893426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.893346 6808 factory.go:223] Registration of the crio container factory successfully Oct 30 21:28:57.893426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.893407 6808 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:28:57.893536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.893430 6808 factory.go:103] Registering Raw factory Oct 30 21:28:57.893536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.893446 6808 manager.go:1199] Started watching for new ooms in manager Oct 30 21:28:57.895682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.895665 6808 manager.go:319] Starting recovery of all containers Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896113 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f25b78-f285-4deb-a96f-b94c4bc6e9f4" volumeName="kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-tuned" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896166 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136a5087-3017-46af-aa81-15b1a899031a" volumeName="kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896181 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" volumeName="kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896195 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d852c03-e50d-4039-bd52-7bfae081a271" volumeName="kubernetes.io/configmap/8d852c03-e50d-4039-bd52-7bfae081a271-config-volume" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896209 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d852c03-e50d-4039-bd52-7bfae081a271" volumeName="kubernetes.io/secret/8d852c03-e50d-4039-bd52-7bfae081a271-metrics-tls" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896222 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896234 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896248 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136a5087-3017-46af-aa81-15b1a899031a" volumeName="kubernetes.io/projected/136a5087-3017-46af-aa81-15b1a899031a-kube-api-access-wckqp" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896262 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136a5087-3017-46af-aa81-15b1a899031a" volumeName="kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896276 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896289 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4422cee8-aec1-4569-9277-bba73b00a9ad" volumeName="kubernetes.io/projected/4422cee8-aec1-4569-9277-bba73b00a9ad-kube-api-access-hclrn" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896301 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896313 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896326 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896343 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/empty-dir/0328b8ce-272d-4232-8681-ef9e0d3a5274-audit-log" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896355 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/projected/0328b8ce-272d-4232-8681-ef9e0d3a5274-kube-api-access-gcpw5" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896369 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896384 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" volumeName="kubernetes.io/projected/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-kube-api-access-xxz9l" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896396 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" volumeName="kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896408 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9233cc4d-86f9-4890-9464-33c7eefd42a8" volumeName="kubernetes.io/empty-dir/9233cc4d-86f9-4890-9464-33c7eefd42a8-tmp-dir" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896421 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9233cc4d-86f9-4890-9464-33c7eefd42a8" volumeName="kubernetes.io/projected/9233cc4d-86f9-4890-9464-33c7eefd42a8-kube-api-access-ds64h" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896432 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" volumeName="kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896446 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896460 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" volumeName="kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-binary-copy" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896472 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896484 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896495 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896509 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f25b78-f285-4deb-a96f-b94c4bc6e9f4" volumeName="kubernetes.io/projected/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-kube-api-access-v254t" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896522 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896532 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b6c776-a4e5-492f-a017-3c9dfc567d56" volumeName="kubernetes.io/configmap/55b6c776-a4e5-492f-a017-3c9dfc567d56-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896542 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6b899c-d914-4adb-9a7c-230e359b1497" volumeName="kubernetes.io/secret/9e6b899c-d914-4adb-9a7c-230e359b1497-agent-certs" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896579 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" volumeName="kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896593 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896607 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4693396-79bd-410d-86a3-5a61f614d231" volumeName="kubernetes.io/configmap/c4693396-79bd-410d-86a3-5a61f614d231-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896640 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f" volumeName="kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896655 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896666 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4422cee8-aec1-4569-9277-bba73b00a9ad" volumeName="kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-cni-binary-copy" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896680 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75908611-03b2-42df-85e5-128bd14c5a43" volumeName="kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896693 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d852c03-e50d-4039-bd52-7bfae081a271" volumeName="kubernetes.io/projected/8d852c03-e50d-4039-bd52-7bfae081a271-kube-api-access-r8jcz" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896704 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896716 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136a5087-3017-46af-aa81-15b1a899031a" volumeName="kubernetes.io/empty-dir/136a5087-3017-46af-aa81-15b1a899031a-volume-directive-shadow" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896730 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" volumeName="kubernetes.io/empty-dir/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896743 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" volumeName="kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-env-overrides" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896758 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/configmap/b4e22dd8-f90b-4512-9ade-3f3ea3329946-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896769 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896783 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896796 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f" volumeName="kubernetes.io/projected/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-kube-api-access-82bsw" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896808 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="129d99fe-11f5-478f-b470-0908f5a42037" volumeName="kubernetes.io/secret/129d99fe-11f5-478f-b470-0908f5a42037-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896866 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136a5087-3017-46af-aa81-15b1a899031a" volumeName="kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896883 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896894 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" volumeName="kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896908 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75908611-03b2-42df-85e5-128bd14c5a43" volumeName="kubernetes.io/projected/75908611-03b2-42df-85e5-128bd14c5a43-kube-api-access-ng5k6" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896920 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" volumeName="kubernetes.io/projected/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-kube-api-access-lktlq" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896929 6808 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897009 6808 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.896935 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897045 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada" volumeName="kubernetes.io/projected/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-kube-api-access-6ddgd" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897084 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4422cee8-aec1-4569-9277-bba73b00a9ad" volumeName="kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-daemon-config" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897111 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897128 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897145 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4693396-79bd-410d-86a3-5a61f614d231" volumeName="kubernetes.io/empty-dir/c4693396-79bd-410d-86a3-5a61f614d231-data-volume" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897161 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4693396-79bd-410d-86a3-5a61f614d231" volumeName="kubernetes.io/projected/c4693396-79bd-410d-86a3-5a61f614d231-kube-api-access-88c2z" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897179 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f25b78-f285-4deb-a96f-b94c4bc6e9f4" volumeName="kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-tmp" seLinuxMountContext="" Oct 30 21:28:57.897200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897251 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897267 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd38803-1d13-470a-97b1-99bc9262d8f0" volumeName="kubernetes.io/projected/5dd38803-1d13-470a-97b1-99bc9262d8f0-kube-api-access-jxshq" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897280 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d852c03-e50d-4039-bd52-7bfae081a271" volumeName="kubernetes.io/empty-dir/8d852c03-e50d-4039-bd52-7bfae081a271-tmp-dir" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897365 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4693396-79bd-410d-86a3-5a61f614d231" volumeName="kubernetes.io/secret/c4693396-79bd-410d-86a3-5a61f614d231-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897377 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-client-ca-bundle" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897390 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136a5087-3017-46af-aa81-15b1a899031a" volumeName="kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897402 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9f17f5b-18dc-4642-a2bb-d778b0524561" volumeName="kubernetes.io/projected/a9f17f5b-18dc-4642-a2bb-d778b0524561-kube-api-access-trx6c" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897413 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9f17f5b-18dc-4642-a2bb-d778b0524561" volumeName="kubernetes.io/secret/a9f17f5b-18dc-4642-a2bb-d778b0524561-cert" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897425 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" volumeName="kubernetes.io/secret/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897437 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897452 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" volumeName="kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-tls" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897465 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b6c776-a4e5-492f-a017-3c9dfc567d56" volumeName="kubernetes.io/projected/55b6c776-a4e5-492f-a017-3c9dfc567d56-kube-api-access-x5wgs" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897477 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6b899c-d914-4adb-9a7c-230e359b1497" volumeName="kubernetes.io/configmap/9e6b899c-d914-4adb-9a7c-230e359b1497-konnectivity-ca" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897489 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897501 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897512 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3231e522-a00c-4ce7-9285-fe3954981036" volumeName="kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897524 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada" volumeName="kubernetes.io/configmap/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-serviceca" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897538 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897550 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="129d99fe-11f5-478f-b470-0908f5a42037" volumeName="kubernetes.io/projected/129d99fe-11f5-478f-b470-0908f5a42037-kube-api-access-hkwx4" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897562 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f8be18f-c1ce-46ae-b36c-df83e438c101" volumeName="kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897574 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" volumeName="kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897587 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/empty-dir/b4e22dd8-f90b-4512-9ade-3f3ea3329946-tmp" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897599 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f" volumeName="kubernetes.io/configmap/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897613 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0328b8ce-272d-4232-8681-ef9e0d3a5274" volumeName="kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897625 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09c3c33e-a59d-4ae8-abec-366b2effbb07" volumeName="kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897637 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" volumeName="kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897721 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897735 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f" volumeName="kubernetes.io/projected/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-kube-api-access-sbqwg" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897747 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e22dd8-f90b-4512-9ade-3f3ea3329946" volumeName="kubernetes.io/projected/b4e22dd8-f90b-4512-9ade-3f3ea3329946-kube-api-access-gck7r" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897762 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897774 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf47ea60-d472-4933-8e64-3f0a42b5116a" volumeName="kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897786 6808 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f" volumeName="kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897798 6808 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.897805 6808 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:28:57.900583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.899871 6808 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:57.902213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.901643 6808 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:28:57.918873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.918840 6808 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:28:57.920469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.920439 6808 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:28:57.920469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.920468 6808 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:28:57.920615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.920489 6808 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:28:57.920615 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:57.920559 6808 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:28:57.924586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.924566 6808 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:57.936155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.936120 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4" exitCode=0 Oct 30 21:28:57.965042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.965014 6808 generic.go:356] "Generic (PLEG): container finished" podID="40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9" containerID="126480897b8c3ebe863aa9f0fec1f0ee88142f744755c76cfb646fb444bbb435" exitCode=0 Oct 30 21:28:57.974148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.974113 6808 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="6c4587e6fabcd380aeefd0d9523c77d4a62e4c868015f38d2e07b09bf2d26d2e" exitCode=0 Oct 30 21:28:57.974148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.974145 6808 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="82f8d0dfad86ceab135e4ede538c7d08b1dafed3ff92bf9bb9fa0fb9006bdcbf" exitCode=0 Oct 30 21:28:57.974297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.974155 6808 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="10174efb224d79369736b5cf7545bd046c9cf8c41bac0acd96a349d7fe8f5958" exitCode=0 Oct 30 21:28:57.974297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.974166 6808 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="472e936680e595d8dc89b54f2cfaa7ec3bae4cb90f5d3782f41d130a3d74afae" exitCode=0 Oct 30 21:28:57.974297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.974175 6808 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="50f26ec6e19d9bdb4d877116168005e73f426389b40dd7fb73e48a800d98e994" exitCode=0 Oct 30 21:28:57.974297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.974183 6808 generic.go:356] "Generic (PLEG): container finished" podID="89806e93-18c5-43a5-b1ed-0f6eae4f20f3" containerID="a786d57ab57f6980a34b5f58f264e0a06b8598d933261893f75c7b12aa697738" exitCode=0 Oct 30 21:28:57.978275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:57.978247 6808 generic.go:356] "Generic (PLEG): container finished" podID="6670d9ca0a34b37e015f8cc739976c92" containerID="11f3d88fdb7194f9ddb9e4ba3bcba853f12d2063b26006697cf830ee4d69ed11" exitCode=0 Oct 30 21:28:58.009094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.009038 6808 manager.go:324] Recovery completed Oct 30 21:28:58.021536 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:58.021513 6808 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:28:58.031287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031266 6808 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:28:58.031287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031284 6808 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:28:58.031408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031304 6808 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:28:58.031464 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031447 6808 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:28:58.031509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031467 6808 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:28:58.031509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031496 6808 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:28:58.031509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031504 6808 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:28:58.031594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031513 6808 policy_none.go:50] "Start" Oct 30 21:28:58.031594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031526 6808 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:28:58.031594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.031537 6808 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:28:58.032786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.032773 6808 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:28:58.032786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.032785 6808 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:28:58.032845 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.032791 6808 policy_none.go:44] "Start" Oct 30 21:28:58.038766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.038735 6808 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:28:58.042489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.042474 6808 manager.go:341] "Starting Device Plugin manager" Oct 30 21:28:58.043039 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:58.042504 6808 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:28:58.043039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043041 6808 server.go:87] "Starting device plugin registration server" Oct 30 21:28:58.043298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043287 6808 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:28:58.043335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043298 6808 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:28:58.043410 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043390 6808 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:28:58.043521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043489 6808 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:28:58.043588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043534 6808 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:28:58.043588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043541 6808 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:28:58.043688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.043579 6808 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:28:58.043520122Z","UUID":"2e286121-59a8-4a35-96ec-505083c2cad1","Handler":null,"Name":"","Endpoint":""} Oct 30 21:28:58.044812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.044785 6808 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:28:58.044920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.044847 6808 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:28:58.045675 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:58.045654 6808 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:28:58.144016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.143980 6808 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:28:58.146042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.146016 6808 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:28:58.146134 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.146050 6808 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:28:58.146134 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.146062 6808 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-168.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:28:58.146134 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.146104 6808 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.162185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.162166 6808 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.162251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.162229 6808 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.222059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.221994 6808 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal"] Oct 30 21:28:58.222618 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222577 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"de6e1dbd7017e381fdeb008c14987499","Type":"ContainerStarted","Data":"ff485c1363d15c4e1713098883e2d9a68e030aff7840656de898156cdd9847b1"} Oct 30 21:28:58.222683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222622 6808 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.222759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222702 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"de6e1dbd7017e381fdeb008c14987499","Type":"ContainerStarted","Data":"c99218bf0e0bacd709121fa56ef950fe2decfbdc682705596378dba15ed7dec6"} Oct 30 21:28:58.222839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222824 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"6670d9ca0a34b37e015f8cc739976c92","Type":"ContainerStarted","Data":"74c2e8c3f9ec0038149ca4436bf76173d2d83130ad98feeeff134a68f347db0b"} Oct 30 21:28:58.222873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222827 6808 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.222873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222850 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"6670d9ca0a34b37e015f8cc739976c92","Type":"ContainerDied","Data":"11f3d88fdb7194f9ddb9e4ba3bcba853f12d2063b26006697cf830ee4d69ed11"} Oct 30 21:28:58.222873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.222870 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" event={"ID":"6670d9ca0a34b37e015f8cc739976c92","Type":"ContainerStarted","Data":"f09d159a00898ce7def24c500eb98973e5fafff870f4ad4f34e5c905d1350d1e"} Oct 30 21:28:58.240352 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.240335 6808 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:58.240390 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:58.240370 6808 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.241106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.241089 6808 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:58.241141 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:58.241134 6808 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.304492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.304422 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de6e1dbd7017e381fdeb008c14987499-config\") pod \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"de6e1dbd7017e381fdeb008c14987499\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.304576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.304516 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.304576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.304540 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.405040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.404999 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.405040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.405044 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.405188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.405064 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de6e1dbd7017e381fdeb008c14987499-config\") pod \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"de6e1dbd7017e381fdeb008c14987499\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.405188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.405107 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/de6e1dbd7017e381fdeb008c14987499-config\") pod \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"de6e1dbd7017e381fdeb008c14987499\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.405188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.405111 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.405188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.405114 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6670d9ca0a34b37e015f8cc739976c92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" (UID: \"6670d9ca0a34b37e015f8cc739976c92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.872801 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.872771 6808 apiserver.go:52] "Watching apiserver" Oct 30 21:28:58.884062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.884043 6808 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:28:58.884666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.884646 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8btfx","kube-system/konnectivity-agent-mcglb","openshift-console/console-546ff4d698-mbksn","openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z","openshift-image-registry/node-ca-8l22v","openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf","openshift-network-operator/iptables-alerter-bc4bx","openshift-multus/network-metrics-daemon-p2nkx","openshift-dns/node-resolver-r7866","openshift-insights/insights-runtime-extractor-8lm4w","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-mq6jf","openshift-network-diagnostics/network-check-target-sq9rx","openshift-dns/dns-default-xfdq8","openshift-monitoring/metrics-server-bdd5667d8-qdt5r","openshift-monitoring/thanos-querier-6f995ff688-xs5n7","openshift-multus/multus-nx794","openshift-monitoring/node-exporter-8psws","openshift-multus/multus-additional-cni-plugins-87ql4","openshift-ovn-kubernetes/ovnkube-node-wnjz7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm","openshift-cluster-node-tuning-operator/tuned-vnxqg"] Oct 30 21:28:58.891855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.891829 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-flwfj\"" Oct 30 21:28:58.892018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.891998 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.892256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.892240 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:28:58.892519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.892496 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:28:58.892991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.892971 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:28:58.893079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893061 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:28:58.893137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893120 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ct8zc\"" Oct 30 21:28:58.893491 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893459 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:28:58.893683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893663 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:28:58.893755 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893693 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99k6q\"" Oct 30 21:28:58.893755 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893702 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.893941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893919 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4rkh\"" Oct 30 21:28:58.894087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.893389 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:28:58.894087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.894055 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpdb2\"" Oct 30 21:28:58.894501 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.894483 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.895012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.894993 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.895147 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895131 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.895225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895210 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.895284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895248 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.895531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895514 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.895639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895621 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.895731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895715 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:28:58.895783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895722 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.895858 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.895524 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.897403 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897382 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.897499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897408 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.897499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897421 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.897499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897458 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:28:58.897657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897551 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.897657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897562 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:28:58.897657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897596 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:28:58.897657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897623 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.897657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897641 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:28:58.897874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897755 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:28:58.897874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897776 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:28:58.897874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.897816 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5jcj9\"" Oct 30 21:28:58.899225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.899209 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8x7v\"" Oct 30 21:28:58.899719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.899700 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:28:58.899846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.899772 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.899906 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.899889 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.899940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.899930 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:28:58.900176 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900160 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.900450 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900392 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:28:58.900450 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900429 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:28:58.900450 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900429 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:28:58.900625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900567 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:28:58.900625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900598 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wf4q\"" Oct 30 21:28:58.900625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900628 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.900741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.900651 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d996q\"" Oct 30 21:28:58.901484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901458 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:28:58.901484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901469 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r96r5\"" Oct 30 21:28:58.901484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901475 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjjj4\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901509 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxqgz\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901512 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pqqf8\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901566 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901570 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pcx6\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901590 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901608 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcj4q\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901608 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901616 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g2c5m\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901615 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:28:58.901692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901667 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:28:58.901979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.901706 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:28:58.902031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.902011 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:28:58.902031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.902013 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z4fj8\"" Oct 30 21:28:58.902122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.902034 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9vdcb\"" Oct 30 21:28:58.902122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.902035 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:28:58.902122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.902103 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:28:58.903938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.903910 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:28:58.904081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.903991 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-48dkn\"" Oct 30 21:28:58.904081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904002 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:28:58.904081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.903992 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:28:58.904081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.903998 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:28:58.904081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904028 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:28:58.904081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904029 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:28:58.904291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904113 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:28:58.904291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904120 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:28:58.904291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904137 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:28:58.904291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904200 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:28:58.904291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904244 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:28:58.904927 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.904910 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:28:58.906642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906622 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b6c776-a4e5-492f-a017-3c9dfc567d56-iptables-alerter-script\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:58.906702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906649 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d852c03-e50d-4039-bd52-7bfae081a271-metrics-tls\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.906702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906665 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-cni-binary-copy\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.906702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906692 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-socket-dir-parent\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.906807 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906762 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-multus\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.906912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906895 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-kubelet\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906919 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d852c03-e50d-4039-bd52-7bfae081a271-metrics-tls\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.907000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906928 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-node-log\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906971 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-script-lib\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.906998 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-multus-certs\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.907139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907003 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b6c776-a4e5-492f-a017-3c9dfc567d56-iptables-alerter-script\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:58.907139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907034 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4693396-79bd-410d-86a3-5a61f614d231-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.907139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907077 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-cni-binary-copy\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.907139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907127 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907245 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-host\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907259 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4693396-79bd-410d-86a3-5a61f614d231-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907280 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907291 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-script-lib\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907320 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907347 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907348 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-ovn\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907369 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907386 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907498 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9233cc4d-86f9-4890-9464-33c7eefd42a8-hosts-file\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907520 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e6b899c-d914-4adb-9a7c-230e359b1497-agent-certs\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907536 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-hostroot\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907543 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907550 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-conf-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907586 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-daemon-config\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.907623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907611 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-systemd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907637 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-log-socket\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907660 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-bin\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907696 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-config\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907724 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wgs\" (UniqueName: \"kubernetes.io/projected/55b6c776-a4e5-492f-a017-3c9dfc567d56-kube-api-access-x5wgs\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907737 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e6b899c-d914-4adb-9a7c-230e359b1497-agent-certs\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907749 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d852c03-e50d-4039-bd52-7bfae081a271-config-volume\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.907834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907801 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-daemon-config\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907895 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovnkube-config\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907919 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d852c03-e50d-4039-bd52-7bfae081a271-config-volume\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.907973 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-bin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908002 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-serviceca\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908024 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddgd\" (UniqueName: \"kubernetes.io/projected/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-kube-api-access-6ddgd\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908054 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5k6\" (UniqueName: \"kubernetes.io/projected/75908611-03b2-42df-85e5-128bd14c5a43-kube-api-access-ng5k6\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908072 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9233cc4d-86f9-4890-9464-33c7eefd42a8-tmp-dir\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908086 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-sys\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.908103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908100 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b899c-d914-4adb-9a7c-230e359b1497-konnectivity-ca\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908114 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclrn\" (UniqueName: \"kubernetes.io/projected/4422cee8-aec1-4569-9277-bba73b00a9ad-kube-api-access-hclrn\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908130 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-etc-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908163 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-netd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908178 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908193 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-textfile\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908208 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-system-cni-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908222 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cnibin\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908237 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908250 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908270 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-netns\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908287 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4693396-79bd-410d-86a3-5a61f614d231-data-volume\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.908341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908303 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908345 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908362 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908378 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908580 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908577 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b899c-d914-4adb-9a7c-230e359b1497-konnectivity-ca\") pod \"konnectivity-agent-mcglb\" (UID: \"9e6b899c-d914-4adb-9a7c-230e359b1497\") " pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908648 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-device-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.908660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908656 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-textfile\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.908850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908672 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-serviceca\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:58.908850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908681 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-var-lib-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.908850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908709 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4693396-79bd-410d-86a3-5a61f614d231-data-volume\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.908850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908715 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.908850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908786 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9233cc4d-86f9-4890-9464-33c7eefd42a8-tmp-dir\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:58.908850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908842 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.909059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908874 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-kubelet-config\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:58.909059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908877 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.909059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908896 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds64h\" (UniqueName: \"kubernetes.io/projected/9233cc4d-86f9-4890-9464-33c7eefd42a8-kube-api-access-ds64h\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:58.909059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908925 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.909059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.908971 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909101 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d852c03-e50d-4039-bd52-7bfae081a271-tmp-dir\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.909298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909161 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-k8s-cni-cncf-io\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.909298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909179 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d852c03-e50d-4039-bd52-7bfae081a271-tmp-dir\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.909298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909208 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-slash\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.909298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909231 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88c2z\" (UniqueName: \"kubernetes.io/projected/c4693396-79bd-410d-86a3-5a61f614d231-kube-api-access-88c2z\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.909298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909234 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.909523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909360 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-modprobe-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909427 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-sys\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909458 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-tmp\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909489 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-cnibin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.909676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909529 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-tmp\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909552 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-host\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:58.909676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909580 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:58.909676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909598 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-metrics-client-ca\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.909676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909663 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.909839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909696 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.909839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909719 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysconfig\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909765 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-kubernetes\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.909839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909798 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b6c776-a4e5-492f-a017-3c9dfc567d56-host-slash\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:58.909839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909819 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75908611-03b2-42df-85e5-128bd14c5a43-metrics-certs\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:58.909839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909834 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-kubelet\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909872 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-etc-kubernetes\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909883 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909899 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-netns\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909933 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqwg\" (UniqueName: \"kubernetes.io/projected/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-kube-api-access-sbqwg\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.909980 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4693396-79bd-410d-86a3-5a61f614d231-crio-socket\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910001 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-metrics-client-ca\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910063 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-dbus\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910084 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-wtmp\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910119 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910139 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f17f5b-18dc-4642-a2bb-d778b0524561-cert\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910155 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910194 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktlq\" (UniqueName: \"kubernetes.io/projected/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-kube-api-access-lktlq\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910216 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v254t\" (UniqueName: \"kubernetes.io/projected/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-kube-api-access-v254t\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.910320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910306 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.910606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910417 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f17f5b-18dc-4642-a2bb-d778b0524561-cert\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:58.910606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910522 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bsw\" (UniqueName: \"kubernetes.io/projected/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-kube-api-access-82bsw\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.910606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910550 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-systemd\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.910606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910572 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-run\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.910789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910608 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-lib-modules\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.910789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910646 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-tuned\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.910789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910683 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4693396-79bd-410d-86a3-5a61f614d231-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.910789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910709 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-tuned\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.910789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910710 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-root\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.910789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910746 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxz9l\" (UniqueName: \"kubernetes.io/projected/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-kube-api-access-xxz9l\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910878 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-conf\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910909 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-var-lib-kubelet\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910926 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxshq\" (UniqueName: \"kubernetes.io/projected/5dd38803-1d13-470a-97b1-99bc9262d8f0-kube-api-access-jxshq\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910932 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4693396-79bd-410d-86a3-5a61f614d231-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.910943 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trx6c\" (UniqueName: \"kubernetes.io/projected/a9f17f5b-18dc-4642-a2bb-d778b0524561-kube-api-access-trx6c\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911032 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jcz\" (UniqueName: \"kubernetes.io/projected/8d852c03-e50d-4039-bd52-7bfae081a271-kube-api-access-r8jcz\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.911063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911054 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-system-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.911303 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911095 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.911303 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911200 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-systemd-units\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.911303 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911226 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.911303 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911254 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.911493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911311 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-env-overrides\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.911493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911345 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovn-node-metrics-cert\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.911493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911372 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:58.911493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911397 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-tls\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.911493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911421 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.911493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911472 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-env-overrides\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.911663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911604 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.911663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911619 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-tls\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.911663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911608 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-ovn-node-metrics-cert\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.911744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.911689 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f8be18f-c1ce-46ae-b36c-df83e438c101-original-pull-secret\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:58.920170 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.920152 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wgs\" (UniqueName: \"kubernetes.io/projected/55b6c776-a4e5-492f-a017-3c9dfc567d56-kube-api-access-x5wgs\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:58.930668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.930651 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclrn\" (UniqueName: \"kubernetes.io/projected/4422cee8-aec1-4569-9277-bba73b00a9ad-kube-api-access-hclrn\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:58.937119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.937098 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5k6\" (UniqueName: \"kubernetes.io/projected/75908611-03b2-42df-85e5-128bd14c5a43-kube-api-access-ng5k6\") pod \"network-metrics-daemon-p2nkx\" (UID: \"75908611-03b2-42df-85e5-128bd14c5a43\") " pod="openshift-multus/network-metrics-daemon-p2nkx" Oct 30 21:28:58.937865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.937849 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bsw\" (UniqueName: \"kubernetes.io/projected/cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f-kube-api-access-82bsw\") pod \"openshift-state-metrics-7bd7489957-csz4z\" (UID: \"cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-csz4z" Oct 30 21:28:58.938742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.938724 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds64h\" (UniqueName: \"kubernetes.io/projected/9233cc4d-86f9-4890-9464-33c7eefd42a8-kube-api-access-ds64h\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:58.942005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.941990 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktlq\" (UniqueName: \"kubernetes.io/projected/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-kube-api-access-lktlq\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:58.942942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.942925 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddgd\" (UniqueName: \"kubernetes.io/projected/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-kube-api-access-6ddgd\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:58.943962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.943933 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqwg\" (UniqueName: \"kubernetes.io/projected/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-kube-api-access-sbqwg\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:58.944572 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.944556 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88c2z\" (UniqueName: \"kubernetes.io/projected/c4693396-79bd-410d-86a3-5a61f614d231-kube-api-access-88c2z\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:58.958151 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.958120 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5j5r\" (UniqueName: \"kubernetes.io/projected/09c3c33e-a59d-4ae8-abec-366b2effbb07-kube-api-access-d5j5r\") pod \"network-check-target-sq9rx\" (UID: \"09c3c33e-a59d-4ae8-abec-366b2effbb07\") " pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:28:58.958384 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.958360 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trx6c\" (UniqueName: \"kubernetes.io/projected/a9f17f5b-18dc-4642-a2bb-d778b0524561-kube-api-access-trx6c\") pod \"ingress-canary-8btfx\" (UID: \"a9f17f5b-18dc-4642-a2bb-d778b0524561\") " pod="openshift-ingress-canary/ingress-canary-8btfx" Oct 30 21:28:58.962054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.962036 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jcz\" (UniqueName: \"kubernetes.io/projected/8d852c03-e50d-4039-bd52-7bfae081a271-kube-api-access-r8jcz\") pod \"dns-default-xfdq8\" (UID: \"8d852c03-e50d-4039-bd52-7bfae081a271\") " pod="openshift-dns/dns-default-xfdq8" Oct 30 21:28:58.963383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.963362 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxz9l\" (UniqueName: \"kubernetes.io/projected/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-kube-api-access-xxz9l\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:58.965852 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.965835 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxshq\" (UniqueName: \"kubernetes.io/projected/5dd38803-1d13-470a-97b1-99bc9262d8f0-kube-api-access-jxshq\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:58.974154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.974073 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v254t\" (UniqueName: \"kubernetes.io/projected/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-kube-api-access-v254t\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:58.991670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.991645 6808 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.992194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.992179 6808 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:58.994371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994341 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:28:58.994464 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994415 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:28:58.994551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994539 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pgqtn\"" Oct 30 21:28:58.994601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994554 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:28:58.994601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994583 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d2jh8\"" Oct 30 21:28:58.994681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994591 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:28:58.994681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994642 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ek4vn38sae3e9\"" Oct 30 21:28:58.994766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994714 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mgk6x\"" Oct 30 21:28:58.994766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.994758 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aahe206i1kld1\"" Oct 30 21:28:58.995020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:58.995001 6808 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:28:59.008043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.008014 6808 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:59.008164 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.008059 6808 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:59.011111 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.011087 6808 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:59.011210 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.011126 6808 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-168.us-east-2.compute.internal" Oct 30 21:28:59.012331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012311 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-multus\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.012405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012340 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-kubelet\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.012405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012359 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-client-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.012405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012374 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.012405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012390 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012411 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-multus-certs\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012415 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-multus\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012440 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-kubelet\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012457 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-multus-certs\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.012475 6808 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.012490 6808 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.012540 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.512519935 +0000 UTC m=+1.708211081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-config" not registered Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012558 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.012593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012584 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckqp\" (UniqueName: \"kubernetes.io/projected/136a5087-3017-46af-aa81-15b1a899031a-kube-api-access-wckqp\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012608 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012631 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-client-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.012635 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.512613125 +0000 UTC m=+1.708304237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012661 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012689 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9233cc4d-86f9-4890-9464-33c7eefd42a8-hosts-file\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012730 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012758 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-log-socket\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012785 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9233cc4d-86f9-4890-9464-33c7eefd42a8-hosts-file\") pod \"node-resolver-r7866\" (UID: \"9233cc4d-86f9-4890-9464-33c7eefd42a8\") " pod="openshift-dns/node-resolver-r7866" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012796 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-bin\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012803 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012850 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-kubernetes\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012858 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012853 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-log-socket\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012853 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-bin\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012897 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012907 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-kubernetes\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.012924 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-bin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013010 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-cni-bin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013040 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-tls\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013064 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013078 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-systemd\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013094 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013112 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013132 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013136 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013147 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-kubelet-config\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013155 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-systemd\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013169 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/136a5087-3017-46af-aa81-15b1a899031a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013201 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013206 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013205 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013238 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/136a5087-3017-46af-aa81-15b1a899031a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013237 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013256 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-kubelet-config\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013282 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-k8s-cni-cncf-io\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013308 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013323 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-run\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013338 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-host\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013381 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-tls\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013386 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada-host\") pod \"node-ca-8l22v\" (UID: \"40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada\") " pod="openshift-image-registry/node-ca-8l22v" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013389 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-run\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.013405 6808 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013407 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013414 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013425 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-k8s-cni-cncf-io\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013432 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013450 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b6c776-a4e5-492f-a017-3c9dfc567d56-host-slash\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.013461 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.513446524 +0000 UTC m=+1.709137626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013480 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b6c776-a4e5-492f-a017-3c9dfc567d56-host-slash\") pod \"iptables-alerter-bc4bx\" (UID: \"55b6c776-a4e5-492f-a017-3c9dfc567d56\") " pod="openshift-network-operator/iptables-alerter-bc4bx" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013488 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-kubelet\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013508 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-var-lib-kubelet\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013513 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-etc-kubernetes\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013538 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-etc-kubernetes\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013539 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-netns\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013572 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-wtmp\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013575 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-netns\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013587 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013608 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013626 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013632 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013652 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-node-exporter-wtmp\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013672 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013703 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013734 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013760 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-var-lib-kubelet\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013788 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013807 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013814 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-systemd-units\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013838 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013850 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013882 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013884 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013924 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-var-lib-kubelet\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013927 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013930 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.013888 6808 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013927 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-systemd-units\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013883 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.013990 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.014011 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.513997838 +0000 UTC m=+1.709688977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:28:59.013970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014010 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014030 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014049 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014069 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4e22dd8-f90b-4512-9ade-3f3ea3329946-metrics-client-ca\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014086 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gck7r\" (UniqueName: \"kubernetes.io/projected/b4e22dd8-f90b-4512-9ade-3f3ea3329946-kube-api-access-gck7r\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014104 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014112 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-socket-dir-parent\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014233 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-socket-dir-parent\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014248 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-node-log\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014252 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4e22dd8-f90b-4512-9ade-3f3ea3329946-metrics-client-ca\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014351 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-node-log\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014360 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-host\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014383 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014386 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014398 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-ovn\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014414 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpw5\" (UniqueName: \"kubernetes.io/projected/0328b8ce-272d-4232-8681-ef9e0d3a5274-kube-api-access-gcpw5\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014436 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-hostroot\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014446 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-ovn\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014460 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-conf-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014438 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-host\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014479 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-hostroot\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014494 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-conf-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014521 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-systemd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014539 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-sys\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014520 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014557 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0328b8ce-272d-4232-8681-ef9e0d3a5274-audit-log\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014592 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-sys\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014592 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-run-systemd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014613 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0328b8ce-272d-4232-8681-ef9e0d3a5274-audit-log\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014595 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cnibin\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014638 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014653 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014658 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-cnibin\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014693 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014699 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014720 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-etc-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.014733 6808 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014736 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-netd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014751 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-system-cni-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.014763 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.514754045 +0000 UTC m=+1.710445171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014778 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89806e93-18c5-43a5-b1ed-0f6eae4f20f3-system-cni-dir\") pod \"multus-additional-cni-plugins-87ql4\" (UID: \"89806e93-18c5-43a5-b1ed-0f6eae4f20f3\") " pod="openshift-multus/multus-additional-cni-plugins-87ql4" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014786 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-modprobe-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014792 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-etc-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014794 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-cni-netd\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014810 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-sys\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014810 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014836 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014865 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-modprobe-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014868 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-sys\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014866 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-netns\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.014898 6808 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014930 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-host-run-netns\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.014973 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-metrics-server-audit-profiles\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015010 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.015020 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.515005972 +0000 UTC m=+1.710697083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"service-ca" not registered Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015051 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015078 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/129d99fe-11f5-478f-b470-0908f5a42037-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015103 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwx4\" (UniqueName: \"kubernetes.io/projected/129d99fe-11f5-478f-b470-0908f5a42037-kube-api-access-hkwx4\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015129 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015180 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-device-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015191 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015206 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-var-lib-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015207 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015216 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-metrics-server-audit-profiles\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015245 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-device-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015270 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-var-lib-openvswitch\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015295 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015317 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-slash\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015318 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/129d99fe-11f5-478f-b470-0908f5a42037-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015340 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015363 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-cnibin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015364 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f-host-slash\") pod \"ovnkube-node-wnjz7\" (UID: \"ad1b40d5-dc2c-4b76-8dcc-b5d16117bd4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015386 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysconfig\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015386 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015406 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-cnibin\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015410 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e22dd8-f90b-4512-9ade-3f3ea3329946-tmp\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015420 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-d\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015436 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-grpc-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015452 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysconfig\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015455 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e22dd8-f90b-4512-9ade-3f3ea3329946-tmp\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015504 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015456 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015538 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4693396-79bd-410d-86a3-5a61f614d231-crio-socket\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015541 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015555 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-dbus\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015572 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015607 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4693396-79bd-410d-86a3-5a61f614d231-crio-socket\") pod \"insights-runtime-extractor-8lm4w\" (UID: \"c4693396-79bd-410d-86a3-5a61f614d231\") " pod="openshift-insights/insights-runtime-extractor-8lm4w" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015623 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f8be18f-c1ce-46ae-b36c-df83e438c101-dbus\") pod \"global-pull-secret-syncer-mq6jf\" (UID: \"5f8be18f-c1ce-46ae-b36c-df83e438c101\") " pod="kube-system/global-pull-secret-syncer-mq6jf" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015643 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4e22dd8-f90b-4512-9ade-3f3ea3329946-secret-grpc-tls\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015646 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015670 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-client-certs\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015674 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5dd38803-1d13-470a-97b1-99bc9262d8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-q4zlm\" (UID: \"5dd38803-1d13-470a-97b1-99bc9262d8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4zlm" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015685 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-lib-modules\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015701 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015720 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/136a5087-3017-46af-aa81-15b1a899031a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015750 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-root\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015771 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-conf\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015796 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-lib-modules\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015830 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0328b8ce-272d-4232-8681-ef9e0d3a5274-secret-metrics-server-client-certs\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015831 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9-root\") pod \"node-exporter-8psws\" (UID: \"40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9\") " pod="openshift-monitoring/node-exporter-8psws" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015841 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-system-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015799 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-system-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015876 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015901 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015916 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015926 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2f25b78-f285-4deb-a96f-b94c4bc6e9f4-etc-sysctl-conf\") pod \"tuned-vnxqg\" (UID: \"d2f25b78-f285-4deb-a96f-b94c4bc6e9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnxqg" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015938 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.015969 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4422cee8-aec1-4569-9277-bba73b00a9ad-multus-cni-dir\") pod \"multus-nx794\" (UID: \"4422cee8-aec1-4569-9277-bba73b00a9ad\") " pod="openshift-multus/multus-nx794" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.016000 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.016884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.016082 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0328b8ce-272d-4232-8681-ef9e0d3a5274-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.025499 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.025482 6808 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:28:59.025499 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.025498 6808 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:28:59.025638 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.025507 6808 projected.go:196] Error preparing data for projected volume kube-api-access-f28vh for pod openshift-console/console-546ff4d698-mbksn: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:28:59.025638 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.025591 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.525578919 +0000 UTC m=+1.721270016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f28vh" (UniqueName: "kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:28:59.026208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.026189 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") pod \"alertmanager-main-0\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:28:59.026518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.026500 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckqp\" (UniqueName: \"kubernetes.io/projected/136a5087-3017-46af-aa81-15b1a899031a-kube-api-access-wckqp\") pod \"kube-state-metrics-5dc8d7d89b-qf7hf\" (UID: \"136a5087-3017-46af-aa81-15b1a899031a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qf7hf" Oct 30 21:28:59.026887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.026866 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpw5\" (UniqueName: \"kubernetes.io/projected/0328b8ce-272d-4232-8681-ef9e0d3a5274-kube-api-access-gcpw5\") pod \"metrics-server-bdd5667d8-qdt5r\" (UID: \"0328b8ce-272d-4232-8681-ef9e0d3a5274\") " pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:28:59.027015 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.026964 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwx4\" (UniqueName: \"kubernetes.io/projected/129d99fe-11f5-478f-b470-0908f5a42037-kube-api-access-hkwx4\") pod \"monitoring-plugin-77f4f76594-zgn2r\" (UID: \"129d99fe-11f5-478f-b470-0908f5a42037\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:28:59.027468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.027450 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gck7r\" (UniqueName: \"kubernetes.io/projected/b4e22dd8-f90b-4512-9ade-3f3ea3329946-kube-api-access-gck7r\") pod \"thanos-querier-6f995ff688-xs5n7\" (UID: \"b4e22dd8-f90b-4512-9ade-3f3ea3329946\") " pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:28:59.519929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.519876 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.519929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.519931 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.520002 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.520019 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520033 6808 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.520069 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520100 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.520085276 +0000 UTC m=+2.715776372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520102 6808 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520128 6808 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.520139 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520154 6808 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520156 6808 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520173 6808 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520159 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.520143225 +0000 UTC m=+2.715834337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"service-ca" not registered Oct 30 21:28:59.520200 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520210 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.520202757 +0000 UTC m=+2.715893854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:28:59.520577 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520224 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.520217568 +0000 UTC m=+2.715908664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-config" not registered Oct 30 21:28:59.520577 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520235 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.520229963 +0000 UTC m=+2.715921060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:28:59.520577 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.520245 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.520240263 +0000 UTC m=+2.715931360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:28:59.620896 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.620859 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:28:59.621058 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.621042 6808 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:28:59.621094 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.621061 6808 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:28:59.621094 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.621071 6808 projected.go:196] Error preparing data for projected volume kube-api-access-f28vh for pod openshift-console/console-546ff4d698-mbksn: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:28:59.621174 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:28:59.621120 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:00.621104481 +0000 UTC m=+2.816795591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28vh" (UniqueName: "kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:28:59.689023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:28:59.688998 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" Oct 30 21:29:00.062665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.062638 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xfdq8" Oct 30 21:29:00.238090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.238066 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zgn2r" Oct 30 21:29:00.528752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.528720 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.528752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.528755 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.528794 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.528835 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528848 6808 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528860 6808 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.528901 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528916 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.528897403 +0000 UTC m=+4.724588515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-config" not registered Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528918 6808 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528927 6808 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:29:00.528970 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528973 6808 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.528989 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.528971678 +0000 UTC m=+4.724662790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.529016 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.529005954 +0000 UTC m=+4.724697054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.529039 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.529029964 +0000 UTC m=+4.724721067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.529058 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.529050429 +0000 UTC m=+4.724741528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.529093 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.529159 6808 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:29:00.529338 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.529196 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.529187081 +0000 UTC m=+4.724878178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"service-ca" not registered Oct 30 21:29:00.629840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:00.629818 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:00.630003 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.629988 6808 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:29:00.630045 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.630008 6808 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:29:00.630045 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.630018 6808 projected.go:196] Error preparing data for projected volume kube-api-access-f28vh for pod openshift-console/console-546ff4d698-mbksn: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:29:00.630105 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:00.630056 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:02.630045586 +0000 UTC m=+4.825736696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28vh" (UniqueName: "kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:29:01.963000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:01.962970 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:29:02.476712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.476685 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" Oct 30 21:29:02.547470 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.547440 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.547612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.547501 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.547612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.547567 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.547732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.547615 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.547732 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.547628 6808 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:29:02.547732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.547688 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.547732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.547720 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.547732 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.547723 6808 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:29:02.548184 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.548156 6808 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:29:02.548308 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.548191 6808 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:29:02.548308 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.548247 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.547760603 +0000 UTC m=+8.743451715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.550044 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.550020448 +0000 UTC m=+8.745711547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-config" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.550090 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.550079219 +0000 UTC m=+8.745770318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.550105 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.550096523 +0000 UTC m=+8.745787666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"service-ca" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.548285 6808 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.550154 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.55014582 +0000 UTC m=+8.745836976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.548408 6808 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:29:02.551966 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.550204 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.550190392 +0000 UTC m=+8.745881536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:29:02.649034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:02.648991 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:02.649154 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.649124 6808 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:29:02.649154 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.649142 6808 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:29:02.649154 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.649152 6808 projected.go:196] Error preparing data for projected volume kube-api-access-f28vh for pod openshift-console/console-546ff4d698-mbksn: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:29:02.649260 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:02.649191 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:06.649179747 +0000 UTC m=+8.844870861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28vh" (UniqueName: "kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:29:05.380918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:05.380891 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:29:05.696339 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:05.696253 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:29:06.583474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.583440 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.583486 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.583521 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.583538 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.583574 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.583593 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583574 6808 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583633 6808 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583601 6808 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583696 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.583672443 +0000 UTC m=+16.779363559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583633 6808 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583725 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.583711635 +0000 UTC m=+16.779402734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583729 6808 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583674 6808 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583743 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.58373455 +0000 UTC m=+16.779425653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"service-ca" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583758 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.583750236 +0000 UTC m=+16.779441337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583782 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.583765247 +0000 UTC m=+16.779456358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-config" not registered Oct 30 21:29:06.583882 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.583843 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.58382722 +0000 UTC m=+16.779518332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:29:06.685061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:06.685038 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:06.685174 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.685162 6808 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:29:06.685209 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.685177 6808 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:29:06.685209 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.685185 6808 projected.go:196] Error preparing data for projected volume kube-api-access-f28vh for pod openshift-console/console-546ff4d698-mbksn: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:29:06.685262 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:06.685222 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh podName:3231e522-a00c-4ce7-9285-fe3954981036 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:14.685210947 +0000 UTC m=+16.880902064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f28vh" (UniqueName: "kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh") pod "console-546ff4d698-mbksn" (UID: "3231e522-a00c-4ce7-9285-fe3954981036") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:29:07.105027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.104968 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:29:07.105543 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.105490 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="alertmanager" containerID="cri-o://0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a" gracePeriod=120 Oct 30 21:29:07.106329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.105847 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy" containerID="cri-o://f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51" gracePeriod=120 Oct 30 21:29:07.106329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.105983 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-web" containerID="cri-o://0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588" gracePeriod=120 Oct 30 21:29:07.106329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.106019 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="prom-label-proxy" containerID="cri-o://e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4" gracePeriod=120 Oct 30 21:29:07.106329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.106148 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="config-reloader" containerID="cri-o://07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af" gracePeriod=120 Oct 30 21:29:07.106329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.106153 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-metric" containerID="cri-o://a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df" gracePeriod=120 Oct 30 21:29:07.575643 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.575615 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:29:07.575832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:07.575819 6808 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:29:08.025260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025228 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4" exitCode=0 Oct 30 21:29:08.025260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025253 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df" exitCode=0 Oct 30 21:29:08.025260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025259 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51" exitCode=0 Oct 30 21:29:08.025260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025265 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af" exitCode=0 Oct 30 21:29:08.025712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025271 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a" exitCode=0 Oct 30 21:29:08.025712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025296 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4"} Oct 30 21:29:08.025712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025354 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df"} Oct 30 21:29:08.025712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025380 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51"} Oct 30 21:29:08.025712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025405 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af"} Oct 30 21:29:08.025712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.025429 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a"} Oct 30 21:29:08.062323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.062300 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bf7567fb-64mnn"] Oct 30 21:29:08.077186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.077166 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.084557 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.084534 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bf7567fb-64mnn"] Oct 30 21:29:08.093985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.093931 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:29:08.094457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.094444 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:29:08.094780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.094769 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:29:08.095477 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.095463 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:29:08.095538 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.095484 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:29:08.095538 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.095521 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:29:08.095600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.095537 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dj9gj\"" Oct 30 21:29:08.099103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.099087 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:29:08.099290 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.099270 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:29:08.196570 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196544 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-oauth-serving-cert\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.196647 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196595 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-serving-cert\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.196723 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196704 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd187cf-c3a9-4109-8c0a-a003faccb300-tmp\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.196759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196750 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-oauth-config\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.196792 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196775 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-trusted-ca-bundle\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.196912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196869 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpq9q\" (UniqueName: \"kubernetes.io/projected/bdd187cf-c3a9-4109-8c0a-a003faccb300-kube-api-access-hpq9q\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.197008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.196990 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-service-ca\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.197057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.197019 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-config\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297443 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd187cf-c3a9-4109-8c0a-a003faccb300-tmp\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297493 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-oauth-config\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297524 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-trusted-ca-bundle\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297577 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpq9q\" (UniqueName: \"kubernetes.io/projected/bdd187cf-c3a9-4109-8c0a-a003faccb300-kube-api-access-hpq9q\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297631 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-service-ca\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297852 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297650 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-config\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297852 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297696 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-oauth-serving-cert\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.297852 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297727 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-serving-cert\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.298032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297867 6808 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:29:08.298032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.297928 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd187cf-c3a9-4109-8c0a-a003faccb300-tmp\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.298600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.298574 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-trusted-ca-bundle\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.298971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.298923 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-service-ca\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.299063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.299036 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-config\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.299124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.298574 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdd187cf-c3a9-4109-8c0a-a003faccb300-oauth-serving-cert\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.300213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.300192 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-serving-cert\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.300296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.300208 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdd187cf-c3a9-4109-8c0a-a003faccb300-console-oauth-config\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.311550 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.311528 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpq9q\" (UniqueName: \"kubernetes.io/projected/bdd187cf-c3a9-4109-8c0a-a003faccb300-kube-api-access-hpq9q\") pod \"console-55bf7567fb-64mnn\" (UID: \"bdd187cf-c3a9-4109-8c0a-a003faccb300\") " pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.365587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.365567 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:08.397014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.396993 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:08.499369 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499337 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499382 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499406 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499439 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499499 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499539 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499566 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499613 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499644 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499681 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.499749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499730 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.500093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499767 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.500093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499822 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bf47ea60-d472-4933-8e64-3f0a42b5116a\" (UID: \"bf47ea60-d472-4933-8e64-3f0a42b5116a\") " Oct 30 21:29:08.500093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.499852 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:08.500252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.500219 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:08.500617 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.500570 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:08.501023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.501005 6808 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-main-db\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.502840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.502812 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.502932 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.502884 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:08.503108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.502994 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:08.503108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.502996 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.503108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.503099 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.503214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.503142 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.504072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.504056 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "kube-api-access-f6v7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:08.504431 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.504416 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.507153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.507136 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.512336 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.512319 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config" pod "bf47ea60-d472-4933-8e64-3f0a42b5116a" (UID: "bf47ea60-d472-4933-8e64-3f0a42b5116a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:08.602481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602433 6808 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-web-config\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602453 6808 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-out\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602463 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6v7s\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-kube-api-access-f6v7s\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602473 6808 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602482 6808 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-cluster-tls-config\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602492 6808 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602501 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-metrics-client-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602511 6808 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf47ea60-d472-4933-8e64-3f0a42b5116a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602520 6808 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602530 6808 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602539 6808 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf47ea60-d472-4933-8e64-3f0a42b5116a-secret-alertmanager-main-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.602676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.602548 6808 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf47ea60-d472-4933-8e64-3f0a42b5116a-tls-assets\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:08.683294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:08.683272 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bf7567fb-64mnn"] Oct 30 21:29:09.030166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.030131 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bf7567fb-64mnn" event={"ID":"bdd187cf-c3a9-4109-8c0a-a003faccb300","Type":"ContainerStarted","Data":"f16197f9f8b8aab8608489a6443e0d40e4b277ffef086d0f3a46c0fb25ec10be"} Oct 30 21:29:09.030563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.030191 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bf7567fb-64mnn" event={"ID":"bdd187cf-c3a9-4109-8c0a-a003faccb300","Type":"ContainerStarted","Data":"a76522bdacdee5c30d5ab02f53fd1951eb99ee05577e4c660d6c617cb1d3d31b"} Oct 30 21:29:09.033371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.033343 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerID="0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588" exitCode=0 Oct 30 21:29:09.033483 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.033393 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588"} Oct 30 21:29:09.033483 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.033430 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.033483 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.033459 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf47ea60-d472-4933-8e64-3f0a42b5116a","Type":"ContainerDied","Data":"32a94c505a332ec8116af313fa49d2586dc06556412c624074ee777aa7f279be"} Oct 30 21:29:09.033600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.033506 6808 scope.go:122] "RemoveContainer" containerID="e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4" Oct 30 21:29:09.041424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.041404 6808 scope.go:122] "RemoveContainer" containerID="a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df" Oct 30 21:29:09.048487 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.048465 6808 scope.go:122] "RemoveContainer" containerID="f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51" Oct 30 21:29:09.052286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.052246 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-55bf7567fb-64mnn" podStartSLOduration=1.052235258 podStartE2EDuration="1.052235258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:09.051680223 +0000 UTC m=+11.247371342" watchObservedRunningTime="2026-10-30 21:29:09.052235258 +0000 UTC m=+11.247926369" Oct 30 21:29:09.055437 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.055420 6808 scope.go:122] "RemoveContainer" containerID="0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588" Oct 30 21:29:09.062031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.062015 6808 scope.go:122] "RemoveContainer" containerID="07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af" Oct 30 21:29:09.068153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.068133 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:29:09.069339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.069319 6808 scope.go:122] "RemoveContainer" containerID="0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a" Oct 30 21:29:09.072714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.072693 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:29:09.076470 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.076451 6808 scope.go:122] "RemoveContainer" containerID="9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4" Oct 30 21:29:09.082996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.082977 6808 scope.go:122] "RemoveContainer" containerID="e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4" Oct 30 21:29:09.083274 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.083257 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4\": container with ID starting with e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4 not found: ID does not exist" containerID="e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4" Oct 30 21:29:09.083317 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.083284 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4"} err="failed to get container status \"e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4\": rpc error: code = NotFound desc = could not find container \"e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4\": container with ID starting with e26ae6c9b2fc43ed0d93eae7ce850512df78e6347fe61d01eb85a0ab6089a1e4 not found: ID does not exist" Oct 30 21:29:09.083363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.083318 6808 scope.go:122] "RemoveContainer" containerID="a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df" Oct 30 21:29:09.083543 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.083528 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df\": container with ID starting with a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df not found: ID does not exist" containerID="a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df" Oct 30 21:29:09.083589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.083550 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df"} err="failed to get container status \"a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df\": rpc error: code = NotFound desc = could not find container \"a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df\": container with ID starting with a65993c5a227c5c7643866eef5b0c01d7c2a059e57af8d113cafffa03d5b06df not found: ID does not exist" Oct 30 21:29:09.083589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.083563 6808 scope.go:122] "RemoveContainer" containerID="f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51" Oct 30 21:29:09.083802 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.083789 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51\": container with ID starting with f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51 not found: ID does not exist" containerID="f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51" Oct 30 21:29:09.083843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.083805 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51"} err="failed to get container status \"f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51\": rpc error: code = NotFound desc = could not find container \"f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51\": container with ID starting with f53ab93e5945b32878633b1454bd9788a3728df4fbdf6b9f562331cbd7932b51 not found: ID does not exist" Oct 30 21:29:09.083843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.083815 6808 scope.go:122] "RemoveContainer" containerID="0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588" Oct 30 21:29:09.084055 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.084037 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588\": container with ID starting with 0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588 not found: ID does not exist" containerID="0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588" Oct 30 21:29:09.084123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084060 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588"} err="failed to get container status \"0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588\": rpc error: code = NotFound desc = could not find container \"0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588\": container with ID starting with 0ac59d4c0f504e89c9542f66e21a3c8f7b6b44be533eb534433aafd510016588 not found: ID does not exist" Oct 30 21:29:09.084123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084077 6808 scope.go:122] "RemoveContainer" containerID="07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af" Oct 30 21:29:09.084319 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.084303 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af\": container with ID starting with 07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af not found: ID does not exist" containerID="07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af" Oct 30 21:29:09.084371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084323 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af"} err="failed to get container status \"07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af\": rpc error: code = NotFound desc = could not find container \"07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af\": container with ID starting with 07958e812b4c13a6efe815847bbcc2c542ce94a396d7bfce2613fbc0603d25af not found: ID does not exist" Oct 30 21:29:09.084371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084336 6808 scope.go:122] "RemoveContainer" containerID="0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a" Oct 30 21:29:09.084568 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.084550 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a\": container with ID starting with 0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a not found: ID does not exist" containerID="0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a" Oct 30 21:29:09.084633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084577 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a"} err="failed to get container status \"0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a\": rpc error: code = NotFound desc = could not find container \"0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a\": container with ID starting with 0bbed242a57232fdcf499fd84a7aed589959ddda4c6d78b5ebf034edc4b5812a not found: ID does not exist" Oct 30 21:29:09.084633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084597 6808 scope.go:122] "RemoveContainer" containerID="9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4" Oct 30 21:29:09.084829 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:09.084809 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4\": container with ID starting with 9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4 not found: ID does not exist" containerID="9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4" Oct 30 21:29:09.084866 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.084836 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4"} err="failed to get container status \"9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4\": rpc error: code = NotFound desc = could not find container \"9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4\": container with ID starting with 9648a61a699bcac278ac56e61c7e125ea392c5b133c015f227aabab2509dafc4 not found: ID does not exist" Oct 30 21:29:09.103686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.103667 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:29:09.103968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.103941 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-web" Oct 30 21:29:09.104013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.103969 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-web" Oct 30 21:29:09.104013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.103980 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-metric" Oct 30 21:29:09.104013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.103986 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-metric" Oct 30 21:29:09.104013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.103998 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="prom-label-proxy" Oct 30 21:29:09.104013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104002 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="prom-label-proxy" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104015 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="init-config-reloader" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104020 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="init-config-reloader" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104030 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104034 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104040 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="config-reloader" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104044 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="config-reloader" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104049 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="alertmanager" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104053 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="alertmanager" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104097 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-web" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104104 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy-metric" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104112 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="prom-label-proxy" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104120 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="init-config-reloader" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104126 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="kube-rbac-proxy" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104132 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="config-reloader" Oct 30 21:29:09.104165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.104136 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" containerName="alertmanager" Oct 30 21:29:09.134347 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.134322 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:29:09.134495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.134481 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.143688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143659 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:29:09.143688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143667 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:29:09.143849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143709 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-48dkn\"" Oct 30 21:29:09.143849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143716 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:29:09.143849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143722 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:29:09.143849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143758 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:29:09.143849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143763 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:29:09.143849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143815 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:29:09.144125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.143994 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:29:09.144125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.144043 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:29:09.206727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.206701 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.206841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.206735 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.206841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.206755 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.206910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.206885 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.206941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.206915 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.206990 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-web-config\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207029 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-config-volume\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207073 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207060 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207109 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207129 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-config-out\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207144 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207217 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207160 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.207270 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.207259 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n89\" (UniqueName: \"kubernetes.io/projected/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-kube-api-access-t2n89\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308010 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n89\" (UniqueName: \"kubernetes.io/projected/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-kube-api-access-t2n89\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308056 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308074 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308195 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308283 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308331 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308486 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-web-config\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308582 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308516 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-config-volume\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308582 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308563 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308595 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308622 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-config-out\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308647 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.308686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.308673 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.309447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.309304 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.310000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.309691 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.310615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.310589 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.311168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.311118 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.311549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.311474 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-config-out\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.311549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.311475 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.311700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.311552 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.311700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.311647 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-web-config\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.311989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.311968 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.312243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.312220 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-config-volume\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.312838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.312817 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.312929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.312912 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.321301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.321282 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n89\" (UniqueName: \"kubernetes.io/projected/58f25f80-76a5-4afb-8b92-00ca20f9ebc9-kube-api-access-t2n89\") pod \"alertmanager-main-0\" (UID: \"58f25f80-76a5-4afb-8b92-00ca20f9ebc9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.534921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.534888 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:29:09.856660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.856611 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:29:09.858805 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:29:09.858771 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f25f80_76a5_4afb_8b92_00ca20f9ebc9.slice/crio-f740ab01f3b8e9fb1d5c2b29a1f95a338bf97b2ac1fcc93cc98974d6d801587a WatchSource:0}: Error finding container f740ab01f3b8e9fb1d5c2b29a1f95a338bf97b2ac1fcc93cc98974d6d801587a: Status 404 returned error can't find the container with id f740ab01f3b8e9fb1d5c2b29a1f95a338bf97b2ac1fcc93cc98974d6d801587a Oct 30 21:29:09.925672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:09.925646 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf47ea60-d472-4933-8e64-3f0a42b5116a" path="/var/lib/kubelet/pods/bf47ea60-d472-4933-8e64-3f0a42b5116a/volumes" Oct 30 21:29:10.039164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:10.039140 6808 generic.go:356] "Generic (PLEG): container finished" podID="58f25f80-76a5-4afb-8b92-00ca20f9ebc9" containerID="14e48a52b0fa3859e98c2c4a96f59b78ff4c67ae6a178784e2065a6d3fa7f2ca" exitCode=0 Oct 30 21:29:10.039474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:10.039217 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerDied","Data":"14e48a52b0fa3859e98c2c4a96f59b78ff4c67ae6a178784e2065a6d3fa7f2ca"} Oct 30 21:29:10.039474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:10.039282 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"f740ab01f3b8e9fb1d5c2b29a1f95a338bf97b2ac1fcc93cc98974d6d801587a"} Oct 30 21:29:11.047008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.046963 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"2ddf468d8e911e2cb8c5760927480b22d4949b31acb514578fb2387267482746"} Oct 30 21:29:11.047351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.047081 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"9b8c4cc16252f087f96be2684f714a1405f4cb752f915ab6f97ab2ec97ccf1e7"} Oct 30 21:29:11.047351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.047109 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"14574c76f1983a8090a44114817fb19958bf551406bd1bb902f2cc05c82524e2"} Oct 30 21:29:11.047351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.047132 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"f590c78b1e5beceae9525a8f1e0d2c6b4ed4345fa6901eda48c44e5e85af95e4"} Oct 30 21:29:11.047351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.047155 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"7556d9d3ff293705050ba87c2cf6d2ea3d381a0e6787a06da31d988cb3a4f80d"} Oct 30 21:29:11.047351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.047176 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58f25f80-76a5-4afb-8b92-00ca20f9ebc9","Type":"ContainerStarted","Data":"5c13c91ab18980c01bc62de446e573d10e9267716cec9388ab9edaae1b6c7b37"} Oct 30 21:29:11.080424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:11.080379 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.0722694170000002 podStartE2EDuration="2.080365851s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.096434ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:11.079864809 +0000 UTC m=+13.275555959" watchObservedRunningTime="2026-10-30 21:29:11.080365851 +0000 UTC m=+13.276056967" Oct 30 21:29:14.660110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660079 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660116 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660150 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660291 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660315 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660341 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660364 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660459 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660538 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660608 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660754 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.660968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.660822 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.760748 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.760723 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:14.761321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:14.761307 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"console-546ff4d698-mbksn\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:18.398176 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:18.398143 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:18.398670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:18.398207 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:18.402746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:18.402723 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:19.079727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:19.079697 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55bf7567fb-64mnn" Oct 30 21:29:22.419191 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:22.419163 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mcglb" Oct 30 21:29:22.676728 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:22.676665 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnjz7" Oct 30 21:29:23.921319 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:23.921282 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-546ff4d698-mbksn" podUID="3231e522-a00c-4ce7-9285-fe3954981036" containerName="console" containerID="cri-o://b3a9b1381f88633d04a9b7801c64d9f97cdb3ad0079ef7b20bfef5ae3dad920b" gracePeriod=15 Oct 30 21:29:24.095844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.095809 6808 generic.go:356] "Generic (PLEG): container finished" podID="3231e522-a00c-4ce7-9285-fe3954981036" containerID="b3a9b1381f88633d04a9b7801c64d9f97cdb3ad0079ef7b20bfef5ae3dad920b" exitCode=0 Oct 30 21:29:24.095995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.095880 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ff4d698-mbksn" event={"ID":"3231e522-a00c-4ce7-9285-fe3954981036","Type":"ContainerDied","Data":"b3a9b1381f88633d04a9b7801c64d9f97cdb3ad0079ef7b20bfef5ae3dad920b"} Oct 30 21:29:24.160995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.160973 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:24.329750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329686 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329741 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329776 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329800 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329819 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329848 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329881 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.329929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.329911 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") pod \"3231e522-a00c-4ce7-9285-fe3954981036\" (UID: \"3231e522-a00c-4ce7-9285-fe3954981036\") " Oct 30 21:29:24.330242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.330083 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:24.330242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.330158 6808 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3231e522-a00c-4ce7-9285-fe3954981036-tmp\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.330242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.330212 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:24.330391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.330356 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:24.330447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.330415 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:24.330544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.330489 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:24.332093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.332072 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "kube-api-access-f28vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:24.332250 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.332231 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:24.332333 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.332315 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert" pod "3231e522-a00c-4ce7-9285-fe3954981036" (UID: "3231e522-a00c-4ce7-9285-fe3954981036"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:24.430441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430411 6808 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-console-config\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.430441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430432 6808 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-oauth-config\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.430441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430442 6808 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3231e522-a00c-4ce7-9285-fe3954981036-console-serving-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.430679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430450 6808 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-trusted-ca-bundle\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.430679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430459 6808 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.430679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430467 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28vh\" (UniqueName: \"kubernetes.io/projected/3231e522-a00c-4ce7-9285-fe3954981036-kube-api-access-f28vh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:24.430679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:24.430477 6808 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3231e522-a00c-4ce7-9285-fe3954981036-oauth-serving-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:25.100452 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:25.100424 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ff4d698-mbksn" Oct 30 21:29:25.100819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:25.100423 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ff4d698-mbksn" event={"ID":"3231e522-a00c-4ce7-9285-fe3954981036","Type":"ContainerDied","Data":"75105fd7ab2e3ae1927d26453a8626f4b484fee31c4451dee88497d43e24f656"} Oct 30 21:29:25.100819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:25.100549 6808 scope.go:122] "RemoveContainer" containerID="b3a9b1381f88633d04a9b7801c64d9f97cdb3ad0079ef7b20bfef5ae3dad920b" Oct 30 21:29:25.122878 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:25.122854 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546ff4d698-mbksn"] Oct 30 21:29:25.126681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:25.126655 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546ff4d698-mbksn"] Oct 30 21:29:25.925365 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:25.925331 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3231e522-a00c-4ce7-9285-fe3954981036" path="/var/lib/kubelet/pods/3231e522-a00c-4ce7-9285-fe3954981036/volumes" Oct 30 21:29:31.965304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:29:31.965273 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sq9rx" Oct 30 21:29:35.696854 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:29:35.696822 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:30:05.696679 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:30:05.696639 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:30:35.696674 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:30:35.696642 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:30:59.811923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.811841 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq"] Oct 30 21:30:59.812499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.812339 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3231e522-a00c-4ce7-9285-fe3954981036" containerName="console" Oct 30 21:30:59.812499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.812357 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3231e522-a00c-4ce7-9285-fe3954981036" containerName="console" Oct 30 21:30:59.812499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.812463 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3231e522-a00c-4ce7-9285-fe3954981036" containerName="console" Oct 30 21:30:59.815690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.815670 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w"] Oct 30 21:30:59.815814 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.815794 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:30:59.818930 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.818912 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:30:59.833235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.833214 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:30:59.833510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.833497 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:30:59.834352 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.834317 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wxl9z\"" Oct 30 21:30:59.834453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.834438 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-787g4\"" Oct 30 21:30:59.841601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.841582 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq"] Oct 30 21:30:59.846908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.846867 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w"] Oct 30 21:30:59.991770 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.991746 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:30:59.991912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.991776 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:30:59.991912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.991795 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjfl\" (UniqueName: \"kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:30:59.991912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.991817 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:30:59.992077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.991973 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:30:59.992077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:30:59.992012 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf44\" (UniqueName: \"kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.092711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.092652 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.092711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.092685 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf44\" (UniqueName: \"kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.092711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.092709 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.092976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.092730 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.092976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.092748 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjfl\" (UniqueName: \"kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.092976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.092766 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.093173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.093153 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.093220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.093179 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.093220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.093188 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.093220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.093205 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.105500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.105474 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjfl\" (UniqueName: \"kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.105681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.105649 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf44\" (UniqueName: \"kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.132440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.132419 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:00.138434 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.138417 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:00.400862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.400831 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv"] Oct 30 21:31:00.404318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.404302 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.411700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.411558 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq"] Oct 30 21:31:00.413689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.413666 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:31:00.413807 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.413760 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:31:00.413866 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.413810 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:31:00.416260 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:31:00.415843 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ee1ceb_a360_4c69_98b1_892a1146844e.slice/crio-55bca0dcdc899bf0fafe0b8992b0be45c948349281c6547bd61edbb3a2a92ed2 WatchSource:0}: Error finding container 55bca0dcdc899bf0fafe0b8992b0be45c948349281c6547bd61edbb3a2a92ed2: Status 404 returned error can't find the container with id 55bca0dcdc899bf0fafe0b8992b0be45c948349281c6547bd61edbb3a2a92ed2 Oct 30 21:31:00.416260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.416074 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-747gg\"" Oct 30 21:31:00.416260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.416139 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:31:00.416260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.416179 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:31:00.416260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.416218 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:31:00.418782 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.418766 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w"] Oct 30 21:31:00.420121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.419177 6808 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:31:00.421778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.421761 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv"] Oct 30 21:31:00.424558 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:31:00.424533 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691f40bc_898f_4130_ba0e_56bab56c712d.slice/crio-ac05fdb168c3cd9b85c20d1b5b24febb6e765cdf301b16271e8aef2e9e664aa6 WatchSource:0}: Error finding container ac05fdb168c3cd9b85c20d1b5b24febb6e765cdf301b16271e8aef2e9e664aa6: Status 404 returned error can't find the container with id ac05fdb168c3cd9b85c20d1b5b24febb6e765cdf301b16271e8aef2e9e664aa6 Oct 30 21:31:00.598510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598480 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sxh\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.598684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598519 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.598684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598550 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.598684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598670 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.598837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598767 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.598837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598830 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.598936 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.598854 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699689 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699732 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699783 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699817 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699843 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699876 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sxh\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.699940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.699910 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.700504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.700480 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.702246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.702227 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.702326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.702277 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.702579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.702562 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.702646 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.702610 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.712229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.712210 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sxh\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.712440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.712423 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k4ggv\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:00.719314 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:00.719297 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:01.008570 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:01.008456 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv"] Oct 30 21:31:01.013045 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:31:01.013000 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a23eeb5_404e_422b_9312_917db81b8328.slice/crio-bc13fcd9e238339cc1edbdad51198d1b1ad88ef2f7609d52edf704e055d2ced3 WatchSource:0}: Error finding container bc13fcd9e238339cc1edbdad51198d1b1ad88ef2f7609d52edf704e055d2ced3: Status 404 returned error can't find the container with id bc13fcd9e238339cc1edbdad51198d1b1ad88ef2f7609d52edf704e055d2ced3 Oct 30 21:31:01.428639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:01.428599 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" event={"ID":"1a23eeb5-404e-422b-9312-917db81b8328","Type":"ContainerStarted","Data":"bc13fcd9e238339cc1edbdad51198d1b1ad88ef2f7609d52edf704e055d2ced3"} Oct 30 21:31:01.430122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:01.430062 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" event={"ID":"691f40bc-898f-4130-ba0e-56bab56c712d","Type":"ContainerStarted","Data":"ac05fdb168c3cd9b85c20d1b5b24febb6e765cdf301b16271e8aef2e9e664aa6"} Oct 30 21:31:01.432627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:01.432598 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" event={"ID":"e5ee1ceb-a360-4c69-98b1-892a1146844e","Type":"ContainerStarted","Data":"55bca0dcdc899bf0fafe0b8992b0be45c948349281c6547bd61edbb3a2a92ed2"} Oct 30 21:31:04.675615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:04.675581 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:31:04.676014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:04.675650 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:31:05.461077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:05.460799 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" event={"ID":"1a23eeb5-404e-422b-9312-917db81b8328","Type":"ContainerStarted","Data":"9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470"} Oct 30 21:31:05.461077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:05.460992 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:05.463382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:05.463352 6808 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-k4ggv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:31:05.463523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:05.463405 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" podUID="1a23eeb5-404e-422b-9312-917db81b8328" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:31:05.491478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:05.491422 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" podStartSLOduration=1.8326498789999999 podStartE2EDuration="5.491405477s" totalImagesPullingTime="3.658755598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:05.489597115 +0000 UTC m=+127.685288238" watchObservedRunningTime="2026-10-30 21:31:05.491405477 +0000 UTC m=+127.687096597" Oct 30 21:31:05.702760 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:31:05.702690 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:31:06.466898 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:06.466867 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:31:07.470824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:07.470796 6808 generic.go:356] "Generic (PLEG): container finished" podID="691f40bc-898f-4130-ba0e-56bab56c712d" containerID="1a59900ec8e3a1d2079f9733cc6d0f2fce905d8bbb21467a278115300af5e143" exitCode=0 Oct 30 21:31:07.471208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:07.470888 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" event={"ID":"691f40bc-898f-4130-ba0e-56bab56c712d","Type":"ContainerDied","Data":"1a59900ec8e3a1d2079f9733cc6d0f2fce905d8bbb21467a278115300af5e143"} Oct 30 21:31:07.472436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:07.472405 6808 generic.go:356] "Generic (PLEG): container finished" podID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerID="d42b7e5413006c70bd4e5ae2c5b8c9a6f21f63f485d6a825f5430bbd3bdbc1d9" exitCode=0 Oct 30 21:31:07.472548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:07.472432 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" event={"ID":"e5ee1ceb-a360-4c69-98b1-892a1146844e","Type":"ContainerDied","Data":"d42b7e5413006c70bd4e5ae2c5b8c9a6f21f63f485d6a825f5430bbd3bdbc1d9"} Oct 30 21:31:10.486572 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:10.486501 6808 generic.go:356] "Generic (PLEG): container finished" podID="691f40bc-898f-4130-ba0e-56bab56c712d" containerID="9684419dba5ee7878ccfac6caf0de7e212b6a81ee12a69822e784cf4468a60ba" exitCode=0 Oct 30 21:31:10.486964 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:10.486584 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" event={"ID":"691f40bc-898f-4130-ba0e-56bab56c712d","Type":"ContainerDied","Data":"9684419dba5ee7878ccfac6caf0de7e212b6a81ee12a69822e784cf4468a60ba"} Oct 30 21:31:10.488552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:10.488525 6808 generic.go:356] "Generic (PLEG): container finished" podID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerID="939595fbe582dc960961ce937646cc5d345cbe32fcb282b11004e8635a7f56d0" exitCode=0 Oct 30 21:31:10.488650 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:10.488564 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" event={"ID":"e5ee1ceb-a360-4c69-98b1-892a1146844e","Type":"ContainerDied","Data":"939595fbe582dc960961ce937646cc5d345cbe32fcb282b11004e8635a7f56d0"} Oct 30 21:31:17.520870 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:17.520837 6808 generic.go:356] "Generic (PLEG): container finished" podID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerID="5f78699aaa9d77abd9323f244d44ce2e840306c512b257af6cc870bea6848bf1" exitCode=0 Oct 30 21:31:17.521263 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:17.520896 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" event={"ID":"e5ee1ceb-a360-4c69-98b1-892a1146844e","Type":"ContainerDied","Data":"5f78699aaa9d77abd9323f244d44ce2e840306c512b257af6cc870bea6848bf1"} Oct 30 21:31:17.522861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:17.522839 6808 generic.go:356] "Generic (PLEG): container finished" podID="691f40bc-898f-4130-ba0e-56bab56c712d" containerID="f9b408c56e0ef825c589f59e603a705c8cc34a276e7cd3a28cf6fd10713338c7" exitCode=0 Oct 30 21:31:17.522986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:17.522877 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" event={"ID":"691f40bc-898f-4130-ba0e-56bab56c712d","Type":"ContainerDied","Data":"f9b408c56e0ef825c589f59e603a705c8cc34a276e7cd3a28cf6fd10713338c7"} Oct 30 21:31:18.650472 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.650445 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:18.678681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.678661 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:18.766821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.766794 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util\") pod \"691f40bc-898f-4130-ba0e-56bab56c712d\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " Oct 30 21:31:18.767019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.766831 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util\") pod \"e5ee1ceb-a360-4c69-98b1-892a1146844e\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " Oct 30 21:31:18.767019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.766868 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle\") pod \"e5ee1ceb-a360-4c69-98b1-892a1146844e\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " Oct 30 21:31:18.767019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.766889 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl\" (UniqueName: \"kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl\") pod \"e5ee1ceb-a360-4c69-98b1-892a1146844e\" (UID: \"e5ee1ceb-a360-4c69-98b1-892a1146844e\") " Oct 30 21:31:18.767019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.766963 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle\") pod \"691f40bc-898f-4130-ba0e-56bab56c712d\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " Oct 30 21:31:18.767262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.767050 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44\" (UniqueName: \"kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44\") pod \"691f40bc-898f-4130-ba0e-56bab56c712d\" (UID: \"691f40bc-898f-4130-ba0e-56bab56c712d\") " Oct 30 21:31:18.767382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.767351 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle" pod "e5ee1ceb-a360-4c69-98b1-892a1146844e" (UID: "e5ee1ceb-a360-4c69-98b1-892a1146844e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:18.767696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.767662 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle" pod "691f40bc-898f-4130-ba0e-56bab56c712d" (UID: "691f40bc-898f-4130-ba0e-56bab56c712d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:18.769327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.769298 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44" pod "691f40bc-898f-4130-ba0e-56bab56c712d" (UID: "691f40bc-898f-4130-ba0e-56bab56c712d"). InnerVolumeSpecName "kube-api-access-tgf44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:18.769474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.769451 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl" pod "e5ee1ceb-a360-4c69-98b1-892a1146844e" (UID: "e5ee1ceb-a360-4c69-98b1-892a1146844e"). InnerVolumeSpecName "kube-api-access-qnjfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:18.771235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.771214 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util" pod "691f40bc-898f-4130-ba0e-56bab56c712d" (UID: "691f40bc-898f-4130-ba0e-56bab56c712d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:18.773056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.773028 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util" pod "e5ee1ceb-a360-4c69-98b1-892a1146844e" (UID: "e5ee1ceb-a360-4c69-98b1-892a1146844e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:18.868006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.867934 6808 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-bundle\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.868006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.867972 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjfl\" (UniqueName: \"kubernetes.io/projected/e5ee1ceb-a360-4c69-98b1-892a1146844e-kube-api-access-qnjfl\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.868006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.867987 6808 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-bundle\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.868006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.867999 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgf44\" (UniqueName: \"kubernetes.io/projected/691f40bc-898f-4130-ba0e-56bab56c712d-kube-api-access-tgf44\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.868199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.868012 6808 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/691f40bc-898f-4130-ba0e-56bab56c712d-util\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:18.868199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:18.868025 6808 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5ee1ceb-a360-4c69-98b1-892a1146844e-util\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.531813 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:19.531782 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" Oct 30 21:31:19.532050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:19.531782 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7hq9w" event={"ID":"691f40bc-898f-4130-ba0e-56bab56c712d","Type":"ContainerDied","Data":"ac05fdb168c3cd9b85c20d1b5b24febb6e765cdf301b16271e8aef2e9e664aa6"} Oct 30 21:31:19.532050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:19.531914 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac05fdb168c3cd9b85c20d1b5b24febb6e765cdf301b16271e8aef2e9e664aa6" Oct 30 21:31:19.533753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:19.533722 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" event={"ID":"e5ee1ceb-a360-4c69-98b1-892a1146844e","Type":"ContainerDied","Data":"55bca0dcdc899bf0fafe0b8992b0be45c948349281c6547bd61edbb3a2a92ed2"} Oct 30 21:31:19.533857 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:19.533768 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bca0dcdc899bf0fafe0b8992b0be45c948349281c6547bd61edbb3a2a92ed2" Oct 30 21:31:19.533857 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:19.533771 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ftllq" Oct 30 21:31:35.697135 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:31:35.697093 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:31:41.310961 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.310917 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp"] Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311218 6808 cpu_manager.go:414] "containerMap: removing container" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="extract" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311234 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="extract" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311259 6808 cpu_manager.go:414] "containerMap: removing container" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="util" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311266 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="util" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311282 6808 cpu_manager.go:414] "containerMap: removing container" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="pull" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311290 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="pull" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311303 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="pull" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311309 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="pull" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311327 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="extract" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311332 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="extract" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311340 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="util" Oct 30 21:31:41.311343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311345 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="util" Oct 30 21:31:41.311733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311400 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ee1ceb-a360-4c69-98b1-892a1146844e" containerName="extract" Oct 30 21:31:41.311733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.311409 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="691f40bc-898f-4130-ba0e-56bab56c712d" containerName="extract" Oct 30 21:31:41.313576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.313536 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.346499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.346474 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp"] Oct 30 21:31:41.393804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.393783 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:31:41.394240 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.394226 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:31:41.396165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.396146 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:31:41.396165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.396146 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jbnqm\"" Oct 30 21:31:41.396280 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.396186 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:31:41.396313 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.396306 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:31:41.430669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.430644 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txgd\" (UniqueName: \"kubernetes.io/projected/be5ce777-dd85-42cd-a303-58b060dd092f-kube-api-access-9txgd\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.430752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.430674 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.430796 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.430764 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.430830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.430799 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be5ce777-dd85-42cd-a303-58b060dd092f-tmp\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.430830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.430824 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.430904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.430843 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be5ce777-dd85-42cd-a303-58b060dd092f-manifests\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531247 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531284 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be5ce777-dd85-42cd-a303-58b060dd092f-tmp\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531306 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531324 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be5ce777-dd85-42cd-a303-58b060dd092f-manifests\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531362 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txgd\" (UniqueName: \"kubernetes.io/projected/be5ce777-dd85-42cd-a303-58b060dd092f-kube-api-access-9txgd\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531567 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531378 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.531781 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.531755 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be5ce777-dd85-42cd-a303-58b060dd092f-manifests\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.533710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.533684 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be5ce777-dd85-42cd-a303-58b060dd092f-tmp\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.533821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.533720 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.533821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.533787 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.533908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.533879 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be5ce777-dd85-42cd-a303-58b060dd092f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.543877 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.543854 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txgd\" (UniqueName: \"kubernetes.io/projected/be5ce777-dd85-42cd-a303-58b060dd092f-kube-api-access-9txgd\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-bxrkp\" (UID: \"be5ce777-dd85-42cd-a303-58b060dd092f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.626848 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.626786 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:41.925283 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:31:41.925251 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5ce777_dd85_42cd_a303_58b060dd092f.slice/crio-c5d5f172bafeb54a3b05cae73a03af88421c3d2591e40d60364bff8dba173033 WatchSource:0}: Error finding container c5d5f172bafeb54a3b05cae73a03af88421c3d2591e40d60364bff8dba173033: Status 404 returned error can't find the container with id c5d5f172bafeb54a3b05cae73a03af88421c3d2591e40d60364bff8dba173033 Oct 30 21:31:41.926653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:41.926629 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp"] Oct 30 21:31:42.618379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:42.618344 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" event={"ID":"be5ce777-dd85-42cd-a303-58b060dd092f","Type":"ContainerStarted","Data":"c5d5f172bafeb54a3b05cae73a03af88421c3d2591e40d60364bff8dba173033"} Oct 30 21:31:45.633263 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:45.633230 6808 generic.go:356] "Generic (PLEG): container finished" podID="be5ce777-dd85-42cd-a303-58b060dd092f" containerID="a3aeaf99835d103d0de02ef8ff31a71ffdbf5be85d972984d870f3c53eabfc4a" exitCode=0 Oct 30 21:31:45.633644 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:45.633278 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" event={"ID":"be5ce777-dd85-42cd-a303-58b060dd092f","Type":"ContainerDied","Data":"a3aeaf99835d103d0de02ef8ff31a71ffdbf5be85d972984d870f3c53eabfc4a"} Oct 30 21:31:46.639349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:46.639318 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" event={"ID":"be5ce777-dd85-42cd-a303-58b060dd092f","Type":"ContainerStarted","Data":"c41106bb1c8738ca1d34303e933b890a93c76007e75faae4dcaf8f7ea97b7e7c"} Oct 30 21:31:46.639764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:46.639462 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:31:46.694113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:46.694046 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" podStartSLOduration=1.952759848 podStartE2EDuration="5.694033436s" totalImagesPullingTime="3.293473237s" totalInitContainerRuntime="447.800351ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:46.691781728 +0000 UTC m=+168.887472846" watchObservedRunningTime="2026-10-30 21:31:46.694033436 +0000 UTC m=+168.889724549" Oct 30 21:31:57.645284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:31:57.645251 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-bxrkp" Oct 30 21:32:05.697651 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:32:05.697613 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:32:13.973769 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:13.973683 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd"] Oct 30 21:32:13.977300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:13.977277 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.006991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.006960 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd"] Oct 30 21:32:14.069404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.069373 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:32:14.069697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.069675 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:32:14.069763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.069740 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-65kvc\"" Oct 30 21:32:14.084364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.084334 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbzx\" (UniqueName: \"kubernetes.io/projected/81dc933c-7be6-4b97-836d-511e02294976-kube-api-access-ngbzx\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.084491 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.084392 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81dc933c-7be6-4b97-836d-511e02294976-tmp\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.084491 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.084476 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dc933c-7be6-4b97-836d-511e02294976-tls-certs\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.185073 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.185039 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbzx\" (UniqueName: \"kubernetes.io/projected/81dc933c-7be6-4b97-836d-511e02294976-kube-api-access-ngbzx\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.185234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.185206 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81dc933c-7be6-4b97-836d-511e02294976-tmp\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.185298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.185249 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dc933c-7be6-4b97-836d-511e02294976-tls-certs\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.185397 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:32:14.185382 6808 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 30 21:32:14.185476 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:32:14.185465 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc933c-7be6-4b97-836d-511e02294976-tls-certs podName:81dc933c-7be6-4b97-836d-511e02294976 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:14.685444615 +0000 UTC m=+196.881135713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/81dc933c-7be6-4b97-836d-511e02294976-tls-certs") pod "kube-auth-proxy-5978df7f68-gjrcd" (UID: "81dc933c-7be6-4b97-836d-511e02294976") : secret "kube-auth-proxy-tls" not found Oct 30 21:32:14.187404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.187375 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81dc933c-7be6-4b97-836d-511e02294976-tmp\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.225419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.225362 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbzx\" (UniqueName: \"kubernetes.io/projected/81dc933c-7be6-4b97-836d-511e02294976-kube-api-access-ngbzx\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.689059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.689024 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dc933c-7be6-4b97-836d-511e02294976-tls-certs\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.691626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.691599 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81dc933c-7be6-4b97-836d-511e02294976-tls-certs\") pod \"kube-auth-proxy-5978df7f68-gjrcd\" (UID: \"81dc933c-7be6-4b97-836d-511e02294976\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:14.894875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:14.894844 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" Oct 30 21:32:15.222783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:15.222714 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd"] Oct 30 21:32:15.225813 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:32:15.225782 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dc933c_7be6_4b97_836d_511e02294976.slice/crio-62c8ccfc05e2b9c1af7d2360361670a73216911f051667a6d50221428c3c7cf3 WatchSource:0}: Error finding container 62c8ccfc05e2b9c1af7d2360361670a73216911f051667a6d50221428c3c7cf3: Status 404 returned error can't find the container with id 62c8ccfc05e2b9c1af7d2360361670a73216911f051667a6d50221428c3c7cf3 Oct 30 21:32:15.755507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:15.755476 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" event={"ID":"81dc933c-7be6-4b97-836d-511e02294976","Type":"ContainerStarted","Data":"62c8ccfc05e2b9c1af7d2360361670a73216911f051667a6d50221428c3c7cf3"} Oct 30 21:32:17.765305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:17.765270 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" event={"ID":"81dc933c-7be6-4b97-836d-511e02294976","Type":"ContainerStarted","Data":"b97cdd75f89b3f7767cd4c73c7b1b7b825eb0342e1423368e34b2b3ee9303778"} Oct 30 21:32:25.128177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.128124 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5978df7f68-gjrcd" podStartSLOduration=9.673570402 podStartE2EDuration="12.128109588s" totalImagesPullingTime="2.454539186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:17.786069867 +0000 UTC m=+199.981760985" watchObservedRunningTime="2026-10-30 21:32:25.128109588 +0000 UTC m=+207.323800706" Oct 30 21:32:25.128571 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.128494 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m"] Oct 30 21:32:25.130690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.130674 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.141279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.141258 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xn5tn\"" Oct 30 21:32:25.143619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.143601 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:32:25.143718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.143706 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:32:25.170058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.170036 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8s64m\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.170145 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.170075 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2dc\" (UniqueName: \"kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc\") pod \"kuadrant-operator-controller-manager-678d87766f-8s64m\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.208577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.208554 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m"] Oct 30 21:32:25.270459 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.270439 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2dc\" (UniqueName: \"kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc\") pod \"kuadrant-operator-controller-manager-678d87766f-8s64m\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.270564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.270499 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8s64m\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.270787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.270772 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8s64m\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.311361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.311341 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2dc\" (UniqueName: \"kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc\") pod \"kuadrant-operator-controller-manager-678d87766f-8s64m\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.442312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.442244 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:25.729055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.728985 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m"] Oct 30 21:32:25.732224 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:32:25.732196 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf264a47c_660d_4427_a489_7218add05706.slice/crio-848da59420d3d89b7670af71ee6a3a165b56137f50f563254dc3b811737f698d WatchSource:0}: Error finding container 848da59420d3d89b7670af71ee6a3a165b56137f50f563254dc3b811737f698d: Status 404 returned error can't find the container with id 848da59420d3d89b7670af71ee6a3a165b56137f50f563254dc3b811737f698d Oct 30 21:32:25.797964 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:25.797923 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" event={"ID":"f264a47c-660d-4427-a489-7218add05706","Type":"ContainerStarted","Data":"848da59420d3d89b7670af71ee6a3a165b56137f50f563254dc3b811737f698d"} Oct 30 21:32:30.822447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:30.822363 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" event={"ID":"f264a47c-660d-4427-a489-7218add05706","Type":"ContainerStarted","Data":"f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99"} Oct 30 21:32:30.822447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:30.822437 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:30.853675 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:30.853630 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" podStartSLOduration=1.117968378 podStartE2EDuration="5.853617937s" totalImagesPullingTime="4.735649559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:30.851512899 +0000 UTC m=+213.047204019" watchObservedRunningTime="2026-10-30 21:32:30.853617937 +0000 UTC m=+213.049309056" Oct 30 21:32:35.696763 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:32:35.696726 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:32:41.828519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:41.828490 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 21:32:59.855420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:59.855389 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcpr5"] Oct 30 21:32:59.865050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:59.865032 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:32:59.868753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:59.868729 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcpr5"] Oct 30 21:32:59.877730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:59.877714 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mqmk5\"" Oct 30 21:32:59.940006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:32:59.939978 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nll2q\" (UniqueName: \"kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q\") pod \"authorino-57f7f8659f-dcpr5\" (UID: \"a3dc63aa-be01-4153-b6cb-2878e5e9118d\") " pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:33:00.041355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.041319 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nll2q\" (UniqueName: \"kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q\") pod \"authorino-57f7f8659f-dcpr5\" (UID: \"a3dc63aa-be01-4153-b6cb-2878e5e9118d\") " pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:33:00.054287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.054260 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nll2q\" (UniqueName: \"kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q\") pod \"authorino-57f7f8659f-dcpr5\" (UID: \"a3dc63aa-be01-4153-b6cb-2878e5e9118d\") " pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:33:00.093117 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.093092 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bjj87"] Oct 30 21:33:00.096649 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.096628 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:33:00.104676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.104654 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bjj87"] Oct 30 21:33:00.187250 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.187228 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:33:00.242925 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.242899 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqbc\" (UniqueName: \"kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc\") pod \"authorino-56c6c9ddd5-bjj87\" (UID: \"329d08d1-37b0-4eaa-af23-9f670c3364ed\") " pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:33:00.344206 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.344178 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqbc\" (UniqueName: \"kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc\") pod \"authorino-56c6c9ddd5-bjj87\" (UID: \"329d08d1-37b0-4eaa-af23-9f670c3364ed\") " pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:33:00.366008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.365981 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqbc\" (UniqueName: \"kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc\") pod \"authorino-56c6c9ddd5-bjj87\" (UID: \"329d08d1-37b0-4eaa-af23-9f670c3364ed\") " pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:33:00.408965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.408932 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:33:00.460187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.460114 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcpr5"] Oct 30 21:33:00.467004 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:33:00.466940 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3dc63aa_be01_4153_b6cb_2878e5e9118d.slice/crio-298cc4ddaf48e906addd13996a196e0a26a8affe2e8778758fcf0183e0c30675 WatchSource:0}: Error finding container 298cc4ddaf48e906addd13996a196e0a26a8affe2e8778758fcf0183e0c30675: Status 404 returned error can't find the container with id 298cc4ddaf48e906addd13996a196e0a26a8affe2e8778758fcf0183e0c30675 Oct 30 21:33:00.677878 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.677851 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bjj87"] Oct 30 21:33:00.680489 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:33:00.680465 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod329d08d1_37b0_4eaa_af23_9f670c3364ed.slice/crio-76ef8853ef51dc5e2ef20a81bcc7a52bd58975f8cdf72d375402e6d6c33cdc8e WatchSource:0}: Error finding container 76ef8853ef51dc5e2ef20a81bcc7a52bd58975f8cdf72d375402e6d6c33cdc8e: Status 404 returned error can't find the container with id 76ef8853ef51dc5e2ef20a81bcc7a52bd58975f8cdf72d375402e6d6c33cdc8e Oct 30 21:33:00.947914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.947881 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" event={"ID":"a3dc63aa-be01-4153-b6cb-2878e5e9118d","Type":"ContainerStarted","Data":"298cc4ddaf48e906addd13996a196e0a26a8affe2e8778758fcf0183e0c30675"} Oct 30 21:33:00.949063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:00.949041 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" event={"ID":"329d08d1-37b0-4eaa-af23-9f670c3364ed","Type":"ContainerStarted","Data":"76ef8853ef51dc5e2ef20a81bcc7a52bd58975f8cdf72d375402e6d6c33cdc8e"} Oct 30 21:33:02.960496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:02.960410 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" event={"ID":"329d08d1-37b0-4eaa-af23-9f670c3364ed","Type":"ContainerStarted","Data":"e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab"} Oct 30 21:33:02.962016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:02.961991 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" event={"ID":"a3dc63aa-be01-4153-b6cb-2878e5e9118d","Type":"ContainerStarted","Data":"d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d"} Oct 30 21:33:02.986481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:02.986425 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" podStartSLOduration=0.953584655 podStartE2EDuration="2.986409258s" totalImagesPullingTime="2.032824603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:02.986098352 +0000 UTC m=+245.181789471" watchObservedRunningTime="2026-10-30 21:33:02.986409258 +0000 UTC m=+245.182100378" Oct 30 21:33:03.009111 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:03.008042 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" podStartSLOduration=1.8336465629999998 podStartE2EDuration="4.008025973s" totalImagesPullingTime="2.17437941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:03.007658674 +0000 UTC m=+245.203349806" watchObservedRunningTime="2026-10-30 21:33:03.008025973 +0000 UTC m=+245.203717093" Oct 30 21:33:03.024613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:03.024582 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcpr5"] Oct 30 21:33:04.977990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:04.977940 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" podUID="a3dc63aa-be01-4153-b6cb-2878e5e9118d" containerName="authorino" containerID="cri-o://d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d" gracePeriod=30 Oct 30 21:33:05.236687 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.236628 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:33:05.392372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.392348 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q\" (UniqueName: \"kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q\") pod \"a3dc63aa-be01-4153-b6cb-2878e5e9118d\" (UID: \"a3dc63aa-be01-4153-b6cb-2878e5e9118d\") " Oct 30 21:33:05.394329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.394301 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q" pod "a3dc63aa-be01-4153-b6cb-2878e5e9118d" (UID: "a3dc63aa-be01-4153-b6cb-2878e5e9118d"). InnerVolumeSpecName "kube-api-access-nll2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:05.493072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.493013 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nll2q\" (UniqueName: \"kubernetes.io/projected/a3dc63aa-be01-4153-b6cb-2878e5e9118d-kube-api-access-nll2q\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:05.697029 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:33:05.696996 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:33:05.982562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.982529 6808 generic.go:356] "Generic (PLEG): container finished" podID="a3dc63aa-be01-4153-b6cb-2878e5e9118d" containerID="d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d" exitCode=0 Oct 30 21:33:05.982994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.982573 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" Oct 30 21:33:05.982994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.982585 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" event={"ID":"a3dc63aa-be01-4153-b6cb-2878e5e9118d","Type":"ContainerDied","Data":"d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d"} Oct 30 21:33:05.982994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.982623 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dcpr5" event={"ID":"a3dc63aa-be01-4153-b6cb-2878e5e9118d","Type":"ContainerDied","Data":"298cc4ddaf48e906addd13996a196e0a26a8affe2e8778758fcf0183e0c30675"} Oct 30 21:33:05.982994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.982646 6808 scope.go:122] "RemoveContainer" containerID="d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d" Oct 30 21:33:05.990626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.990610 6808 scope.go:122] "RemoveContainer" containerID="d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d" Oct 30 21:33:05.990889 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:33:05.990871 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d\": container with ID starting with d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d not found: ID does not exist" containerID="d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d" Oct 30 21:33:05.990965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:05.990897 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d"} err="failed to get container status \"d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d\": rpc error: code = NotFound desc = could not find container \"d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d\": container with ID starting with d5c105cf41afa8b05f2c835017884511d118baa4857f79cac7a911a179ee1d5d not found: ID does not exist" Oct 30 21:33:06.002188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:06.002159 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcpr5"] Oct 30 21:33:06.007164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:06.007145 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dcpr5"] Oct 30 21:33:07.939211 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:07.936657 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dc63aa-be01-4153-b6cb-2878e5e9118d" path="/var/lib/kubelet/pods/a3dc63aa-be01-4153-b6cb-2878e5e9118d/volumes" Oct 30 21:33:15.678099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.678067 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x"] Oct 30 21:33:15.678462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.678420 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a3dc63aa-be01-4153-b6cb-2878e5e9118d" containerName="authorino" Oct 30 21:33:15.678462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.678431 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dc63aa-be01-4153-b6cb-2878e5e9118d" containerName="authorino" Oct 30 21:33:15.678535 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.678484 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dc63aa-be01-4153-b6cb-2878e5e9118d" containerName="authorino" Oct 30 21:33:15.681373 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.681354 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.692195 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.692177 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:33:15.692284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.692202 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:33:15.698435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.698384 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x"] Oct 30 21:33:15.706519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.706499 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zrxrd\"" Oct 30 21:33:15.706620 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.706543 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:33:15.779232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.779204 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljdt\" (UniqueName: \"kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt\") pod \"llmisvc-controller-manager-869d5dcc5d-4js7x\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.779368 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.779274 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4js7x\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.880518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.880459 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljdt\" (UniqueName: \"kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt\") pod \"llmisvc-controller-manager-869d5dcc5d-4js7x\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.880702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.880612 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4js7x\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.883158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.883135 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4js7x\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.896119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.896095 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljdt\" (UniqueName: \"kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt\") pod \"llmisvc-controller-manager-869d5dcc5d-4js7x\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:15.993765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:15.993708 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:16.304943 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.304871 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x"] Oct 30 21:33:16.308105 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:33:16.308079 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod052edc80_445b_483d_b47e_ea9755040cfc.slice/crio-856723885a3213039ec1a56291732d4b6cf89a786d5c46b076a10e5bd6df90e5 WatchSource:0}: Error finding container 856723885a3213039ec1a56291732d4b6cf89a786d5c46b076a10e5bd6df90e5: Status 404 returned error can't find the container with id 856723885a3213039ec1a56291732d4b6cf89a786d5c46b076a10e5bd6df90e5 Oct 30 21:33:16.520694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.520665 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv"] Oct 30 21:33:16.525067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.525052 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.540670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.540649 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv"] Oct 30 21:33:16.687291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687267 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tr6\" (UniqueName: \"kubernetes.io/projected/e971dbae-c1c7-499f-9b1d-79789f4c3769-kube-api-access-26tr6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.687633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687298 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.687633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687327 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971dbae-c1c7-499f-9b1d-79789f4c3769-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.687633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687462 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.687633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687505 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.687633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687529 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.687633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.687585 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.788744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.788713 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.788876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.788763 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tr6\" (UniqueName: \"kubernetes.io/projected/e971dbae-c1c7-499f-9b1d-79789f4c3769-kube-api-access-26tr6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.788876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.788795 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.788876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.788825 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971dbae-c1c7-499f-9b1d-79789f4c3769-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.788876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.788846 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.789122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.788996 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.789122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.789036 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.789544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.789517 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.791379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.791357 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.791475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.791375 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971dbae-c1c7-499f-9b1d-79789f4c3769-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.791574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.791550 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.791701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.791683 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.803220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.803197 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971dbae-c1c7-499f-9b1d-79789f4c3769-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.803380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.803360 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tr6\" (UniqueName: \"kubernetes.io/projected/e971dbae-c1c7-499f-9b1d-79789f4c3769-kube-api-access-26tr6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rc7lv\" (UID: \"e971dbae-c1c7-499f-9b1d-79789f4c3769\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:16.836993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:16.836969 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:17.032806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:17.032715 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" event={"ID":"052edc80-445b-483d-b47e-ea9755040cfc","Type":"ContainerStarted","Data":"856723885a3213039ec1a56291732d4b6cf89a786d5c46b076a10e5bd6df90e5"} Oct 30 21:33:17.159837 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:33:17.159801 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode971dbae_c1c7_499f_9b1d_79789f4c3769.slice/crio-138e332ddef776bf1952b917c0512703199708dc9b4b4e9b75a4e2a72d17e4fd WatchSource:0}: Error finding container 138e332ddef776bf1952b917c0512703199708dc9b4b4e9b75a4e2a72d17e4fd: Status 404 returned error can't find the container with id 138e332ddef776bf1952b917c0512703199708dc9b4b4e9b75a4e2a72d17e4fd Oct 30 21:33:17.162508 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:17.162482 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv"] Oct 30 21:33:17.164079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:17.164046 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:33:17.164185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:17.164101 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:33:18.039407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:18.039320 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" event={"ID":"e971dbae-c1c7-499f-9b1d-79789f4c3769","Type":"ContainerStarted","Data":"4c4ad459a8963987bd32899a5e0dd650624fba5e6caf9cc530c776a70f2b4cdd"} Oct 30 21:33:18.039407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:18.039375 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" event={"ID":"e971dbae-c1c7-499f-9b1d-79789f4c3769","Type":"ContainerStarted","Data":"138e332ddef776bf1952b917c0512703199708dc9b4b4e9b75a4e2a72d17e4fd"} Oct 30 21:33:18.039924 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:18.039559 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:18.041860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:18.041829 6808 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rc7lv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:33:18.042005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:18.041887 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" podUID="e971dbae-c1c7-499f-9b1d-79789f4c3769" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:33:18.072247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:18.072198 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" podStartSLOduration=2.072182684 podStartE2EDuration="2.072182684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:18.07070404 +0000 UTC m=+260.266395161" watchObservedRunningTime="2026-10-30 21:33:18.072182684 +0000 UTC m=+260.267873805" Oct 30 21:33:19.045755 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.045704 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" event={"ID":"052edc80-445b-483d-b47e-ea9755040cfc","Type":"ContainerStarted","Data":"4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75"} Oct 30 21:33:19.046454 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.046434 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rc7lv" Oct 30 21:33:19.066501 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.066449 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" podStartSLOduration=2.169657332 podStartE2EDuration="4.066433735s" totalImagesPullingTime="1.896776403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:19.064601097 +0000 UTC m=+261.260292213" watchObservedRunningTime="2026-10-30 21:33:19.066433735 +0000 UTC m=+261.262124857" Oct 30 21:33:19.124630 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.124600 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv"] Oct 30 21:33:19.124805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.124784 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" podUID="1a23eeb5-404e-422b-9312-917db81b8328" containerName="discovery" containerID="cri-o://9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470" gracePeriod=30 Oct 30 21:33:19.370631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.370604 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:33:19.514725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.514691 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.514918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.514741 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.514918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.514763 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.514918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.514791 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.514918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.514851 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.514918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.514887 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.515221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.515008 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh\") pod \"1a23eeb5-404e-422b-9312-917db81b8328\" (UID: \"1a23eeb5-404e-422b-9312-917db81b8328\") " Oct 30 21:33:19.515221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.515199 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:19.515334 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.515308 6808 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-ca-configmap\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:19.517252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.517224 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:19.517371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.517264 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:19.517553 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.517527 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:19.517629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.517542 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:19.517629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.517553 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:19.517629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.517566 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh" pod "1a23eeb5-404e-422b-9312-917db81b8328" (UID: "1a23eeb5-404e-422b-9312-917db81b8328"). InnerVolumeSpecName "kube-api-access-w5sxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:19.616577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.616493 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:19.616577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.616520 6808 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-cacerts\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:19.616577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.616533 6808 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-csr-dns-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:19.616577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.616548 6808 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a23eeb5-404e-422b-9312-917db81b8328-istio-kubeconfig\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:19.616577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.616560 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5sxh\" (UniqueName: \"kubernetes.io/projected/1a23eeb5-404e-422b-9312-917db81b8328-kube-api-access-w5sxh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:19.616577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:19.616573 6808 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a23eeb5-404e-422b-9312-917db81b8328-local-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:20.052761 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.052720 6808 generic.go:356] "Generic (PLEG): container finished" podID="1a23eeb5-404e-422b-9312-917db81b8328" containerID="9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470" exitCode=0 Oct 30 21:33:20.053363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.052777 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" Oct 30 21:33:20.053363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.052763 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" event={"ID":"1a23eeb5-404e-422b-9312-917db81b8328","Type":"ContainerDied","Data":"9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470"} Oct 30 21:33:20.053363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.052921 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv" event={"ID":"1a23eeb5-404e-422b-9312-917db81b8328","Type":"ContainerDied","Data":"bc13fcd9e238339cc1edbdad51198d1b1ad88ef2f7609d52edf704e055d2ced3"} Oct 30 21:33:20.053363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.052976 6808 scope.go:122] "RemoveContainer" containerID="9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470" Oct 30 21:33:20.053626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.053402 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:20.064382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.064366 6808 scope.go:122] "RemoveContainer" containerID="9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470" Oct 30 21:33:20.064674 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:33:20.064652 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470\": container with ID starting with 9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470 not found: ID does not exist" containerID="9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470" Oct 30 21:33:20.064759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.064686 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470"} err="failed to get container status \"9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470\": rpc error: code = NotFound desc = could not find container \"9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470\": container with ID starting with 9c1521ad88eec7a9bae5a052b2670711ba360a9662c15b91d97034bf130f6470 not found: ID does not exist" Oct 30 21:33:20.074581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.074557 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv"] Oct 30 21:33:20.086414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:20.086218 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k4ggv"] Oct 30 21:33:21.933039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:21.933005 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a23eeb5-404e-422b-9312-917db81b8328" path="/var/lib/kubelet/pods/1a23eeb5-404e-422b-9312-917db81b8328/volumes" Oct 30 21:33:35.697137 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:33:35.697102 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:33:44.950919 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.950840 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85"] Oct 30 21:33:44.951387 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.951226 6808 cpu_manager.go:414] "containerMap: removing container" podUID="1a23eeb5-404e-422b-9312-917db81b8328" containerName="discovery" Oct 30 21:33:44.951387 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.951239 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a23eeb5-404e-422b-9312-917db81b8328" containerName="discovery" Oct 30 21:33:44.951387 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.951293 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a23eeb5-404e-422b-9312-917db81b8328" containerName="discovery" Oct 30 21:33:44.954162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.954145 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:44.963534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.963508 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 21:33:44.963534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.963527 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 21:33:44.963534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.963508 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 21:33:44.963765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.963597 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-bj6t8\"" Oct 30 21:33:44.967868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.967843 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85"] Oct 30 21:33:44.999688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.999660 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4c20eec-664a-4952-9368-7f8e414ce138-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:44.999797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.999703 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c20eec-664a-4952-9368-7f8e414ce138-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:44.999797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.999720 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d4c20eec-664a-4952-9368-7f8e414ce138-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:44.999897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:44.999878 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbw8\" (UniqueName: \"kubernetes.io/projected/d4c20eec-664a-4952-9368-7f8e414ce138-kube-api-access-rtbw8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.100464 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.100442 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbw8\" (UniqueName: \"kubernetes.io/projected/d4c20eec-664a-4952-9368-7f8e414ce138-kube-api-access-rtbw8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.100589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.100484 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4c20eec-664a-4952-9368-7f8e414ce138-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.100589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.100522 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c20eec-664a-4952-9368-7f8e414ce138-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.100589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.100539 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d4c20eec-664a-4952-9368-7f8e414ce138-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.101285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.101265 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c20eec-664a-4952-9368-7f8e414ce138-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.101514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.101497 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d4c20eec-664a-4952-9368-7f8e414ce138-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.102911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.102891 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4c20eec-664a-4952-9368-7f8e414ce138-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.114420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.114393 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbw8\" (UniqueName: \"kubernetes.io/projected/d4c20eec-664a-4952-9368-7f8e414ce138-kube-api-access-rtbw8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcv85\" (UID: \"d4c20eec-664a-4952-9368-7f8e414ce138\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.267451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.267388 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:45.540178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:45.540109 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85"] Oct 30 21:33:45.543894 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:33:45.543864 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c20eec_664a_4952_9368_7f8e414ce138.slice/crio-d817f017c34e6dc4900b690e611af1178eac0bb724d5f541cd08dbb20eef357c WatchSource:0}: Error finding container d817f017c34e6dc4900b690e611af1178eac0bb724d5f541cd08dbb20eef357c: Status 404 returned error can't find the container with id d817f017c34e6dc4900b690e611af1178eac0bb724d5f541cd08dbb20eef357c Oct 30 21:33:46.161411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:46.161382 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" event={"ID":"d4c20eec-664a-4952-9368-7f8e414ce138","Type":"ContainerStarted","Data":"d817f017c34e6dc4900b690e611af1178eac0bb724d5f541cd08dbb20eef357c"} Oct 30 21:33:51.060700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:51.060670 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:33:57.213202 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:57.213166 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" event={"ID":"d4c20eec-664a-4952-9368-7f8e414ce138","Type":"ContainerStarted","Data":"81a648406721a4624097e5affaea6d35dff74ebdca071ccacb48866b23ac40a4"} Oct 30 21:33:57.213551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:57.213244 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:33:57.233770 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:33:57.233615 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" podStartSLOduration=1.667038292 podStartE2EDuration="13.233600891s" totalImagesPullingTime="11.566562599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:57.233113207 +0000 UTC m=+299.428804326" watchObservedRunningTime="2026-10-30 21:33:57.233600891 +0000 UTC m=+299.429292012" Oct 30 21:34:05.697143 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:34:05.697106 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:34:08.222618 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:08.222585 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" Oct 30 21:34:16.799047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.799015 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45"] Oct 30 21:34:16.802305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.802285 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:16.813302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.813281 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:34:16.813482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.813464 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:34:16.813759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.813682 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:34:16.814327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.814307 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:34:16.815678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.815652 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45"] Oct 30 21:34:16.961635 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.961604 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:16.961768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.961687 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:16.961768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.961710 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvxh\" (UniqueName: \"kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:16.961768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:16.961748 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.063098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.063035 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.063098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.063076 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.063281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.063128 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.063281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.063157 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvxh\" (UniqueName: \"kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.063725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.063703 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.065652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.065626 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.065743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.065632 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.082344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.082323 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvxh\" (UniqueName: \"kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh\") pod \"maas-api-6d669c7fcd-7mk45\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.115043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.115021 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:17.398229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:17.398197 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45"] Oct 30 21:34:17.400727 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:34:17.400701 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15df2642_616a_4b7a_a075_e451a5de50cf.slice/crio-a0b291f75a76dcedaac3a978bbe148adf53c34d6a8a7f98d13c553aed96b2c5b WatchSource:0}: Error finding container a0b291f75a76dcedaac3a978bbe148adf53c34d6a8a7f98d13c553aed96b2c5b: Status 404 returned error can't find the container with id a0b291f75a76dcedaac3a978bbe148adf53c34d6a8a7f98d13c553aed96b2c5b Oct 30 21:34:18.314908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:18.314867 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" event={"ID":"15df2642-616a-4b7a-a075-e451a5de50cf","Type":"ContainerStarted","Data":"a0b291f75a76dcedaac3a978bbe148adf53c34d6a8a7f98d13c553aed96b2c5b"} Oct 30 21:34:20.325681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:20.325641 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" event={"ID":"15df2642-616a-4b7a-a075-e451a5de50cf","Type":"ContainerStarted","Data":"5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0"} Oct 30 21:34:20.326061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:20.325748 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:20.346686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:20.346632 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" podStartSLOduration=1.873509795 podStartE2EDuration="4.346617079s" totalImagesPullingTime="2.473107284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:20.345992979 +0000 UTC m=+322.541684097" watchObservedRunningTime="2026-10-30 21:34:20.346617079 +0000 UTC m=+322.542308199" Oct 30 21:34:26.334217 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:26.334186 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:27.177933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.177899 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kdpj6"] Oct 30 21:34:27.357855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.357820 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kdpj6"] Oct 30 21:34:27.357855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.357858 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kdpj6"] Oct 30 21:34:27.358357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.357877 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b9864"] Oct 30 21:34:27.358357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.357985 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kdpj6" Oct 30 21:34:27.363106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.363086 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b9864"] Oct 30 21:34:27.363223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.363164 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.364364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.364346 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kdpj6" Oct 30 21:34:27.367337 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.367320 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:34:27.453871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.453800 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert\") pod \"authorino-6889cbd8c4-b9864\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.453871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.453842 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcq4\" (UniqueName: \"kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4\") pod \"authorino-6889cbd8c4-b9864\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.554333 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.554303 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert\") pod \"authorino-6889cbd8c4-b9864\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.554484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.554351 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcq4\" (UniqueName: \"kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4\") pod \"authorino-6889cbd8c4-b9864\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.556762 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.556741 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert\") pod \"authorino-6889cbd8c4-b9864\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.566738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.566714 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcq4\" (UniqueName: \"kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4\") pod \"authorino-6889cbd8c4-b9864\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.567235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.567216 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b9864"] Oct 30 21:34:27.567385 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.567374 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:27.595188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.595165 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69d7877fb6-r25n9"] Oct 30 21:34:27.621011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.620981 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69d7877fb6-r25n9"] Oct 30 21:34:27.621162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.621080 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.655676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.655598 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert\") pod \"authorino-69d7877fb6-r25n9\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.655676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.655671 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5cvq\" (UniqueName: \"kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq\") pod \"authorino-69d7877fb6-r25n9\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.757035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.756965 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert\") pod \"authorino-69d7877fb6-r25n9\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.757035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.757011 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5cvq\" (UniqueName: \"kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq\") pod \"authorino-69d7877fb6-r25n9\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.759235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.759212 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert\") pod \"authorino-69d7877fb6-r25n9\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.769736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.769713 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5cvq\" (UniqueName: \"kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq\") pod \"authorino-69d7877fb6-r25n9\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:27.868348 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.868323 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b9864"] Oct 30 21:34:27.870315 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:34:27.870290 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc40e17b3_8c78_49f3_80cd_dc793dc80c91.slice/crio-18b49f6e7f046e11258fe7e814ee0cf2020452b6864392f32266585bebaf5e17 WatchSource:0}: Error finding container 18b49f6e7f046e11258fe7e814ee0cf2020452b6864392f32266585bebaf5e17: Status 404 returned error can't find the container with id 18b49f6e7f046e11258fe7e814ee0cf2020452b6864392f32266585bebaf5e17 Oct 30 21:34:27.949377 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:27.949344 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:34:28.248401 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:28.248380 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69d7877fb6-r25n9"] Oct 30 21:34:28.250816 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:34:28.250790 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2056af91_4fda_4711_b425_d85693ee2eac.slice/crio-e6e27dff4ec551b61b04103a62f7e1e2cc72b66b99d3479b1fdfa9e9d509107e WatchSource:0}: Error finding container e6e27dff4ec551b61b04103a62f7e1e2cc72b66b99d3479b1fdfa9e9d509107e: Status 404 returned error can't find the container with id e6e27dff4ec551b61b04103a62f7e1e2cc72b66b99d3479b1fdfa9e9d509107e Oct 30 21:34:28.365234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:28.365166 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b9864" event={"ID":"c40e17b3-8c78-49f3-80cd-dc793dc80c91","Type":"ContainerStarted","Data":"18b49f6e7f046e11258fe7e814ee0cf2020452b6864392f32266585bebaf5e17"} Oct 30 21:34:28.366394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:28.366371 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kdpj6" Oct 30 21:34:28.366486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:28.366388 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69d7877fb6-r25n9" event={"ID":"2056af91-4fda-4711-b425-d85693ee2eac","Type":"ContainerStarted","Data":"e6e27dff4ec551b61b04103a62f7e1e2cc72b66b99d3479b1fdfa9e9d509107e"} Oct 30 21:34:28.397889 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:28.397865 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kdpj6"] Oct 30 21:34:28.404005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:28.403980 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kdpj6"] Oct 30 21:34:29.372601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.372568 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b9864" event={"ID":"c40e17b3-8c78-49f3-80cd-dc793dc80c91","Type":"ContainerStarted","Data":"20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b"} Oct 30 21:34:29.373056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.372614 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-b9864" podUID="c40e17b3-8c78-49f3-80cd-dc793dc80c91" containerName="authorino" containerID="cri-o://20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b" gracePeriod=30 Oct 30 21:34:29.399532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.398090 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-b9864" podStartSLOduration=1.394016012 podStartE2EDuration="2.398071136s" totalImagesPullingTime="1.004055124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:29.396615294 +0000 UTC m=+331.592306415" watchObservedRunningTime="2026-10-30 21:34:29.398071136 +0000 UTC m=+331.593762256" Oct 30 21:34:29.668137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.668097 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:29.775113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.775077 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert\" (UniqueName: \"kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert\") pod \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " Oct 30 21:34:29.775113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.775118 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4\" (UniqueName: \"kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4\") pod \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\" (UID: \"c40e17b3-8c78-49f3-80cd-dc793dc80c91\") " Oct 30 21:34:29.777233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.777204 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4" pod "c40e17b3-8c78-49f3-80cd-dc793dc80c91" (UID: "c40e17b3-8c78-49f3-80cd-dc793dc80c91"). InnerVolumeSpecName "kube-api-access-wqcq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:29.804685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.804640 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert" pod "c40e17b3-8c78-49f3-80cd-dc793dc80c91" (UID: "c40e17b3-8c78-49f3-80cd-dc793dc80c91"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:29.876322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.876240 6808 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c40e17b3-8c78-49f3-80cd-dc793dc80c91-tls-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:29.876322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:29.876270 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcq4\" (UniqueName: \"kubernetes.io/projected/c40e17b3-8c78-49f3-80cd-dc793dc80c91-kube-api-access-wqcq4\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:30.378091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.378055 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69d7877fb6-r25n9" event={"ID":"2056af91-4fda-4711-b425-d85693ee2eac","Type":"ContainerStarted","Data":"757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8"} Oct 30 21:34:30.379327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.379303 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-b9864" Oct 30 21:34:30.379462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.379271 6808 generic.go:356] "Generic (PLEG): container finished" podID="c40e17b3-8c78-49f3-80cd-dc793dc80c91" containerID="20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b" exitCode=0 Oct 30 21:34:30.379462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.379350 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b9864" event={"ID":"c40e17b3-8c78-49f3-80cd-dc793dc80c91","Type":"ContainerDied","Data":"20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b"} Oct 30 21:34:30.379462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.379422 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-b9864" event={"ID":"c40e17b3-8c78-49f3-80cd-dc793dc80c91","Type":"ContainerDied","Data":"18b49f6e7f046e11258fe7e814ee0cf2020452b6864392f32266585bebaf5e17"} Oct 30 21:34:30.379462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.379438 6808 scope.go:122] "RemoveContainer" containerID="20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b" Oct 30 21:34:30.388312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.388294 6808 scope.go:122] "RemoveContainer" containerID="20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b" Oct 30 21:34:30.388589 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:34:30.388567 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b\": container with ID starting with 20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b not found: ID does not exist" containerID="20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b" Oct 30 21:34:30.388705 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.388594 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b"} err="failed to get container status \"20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b\": rpc error: code = NotFound desc = could not find container \"20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b\": container with ID starting with 20b56e28df40aaf55ff45d938cb9c023c8343fd0e5fa781dec8d82a482b6d86b not found: ID does not exist" Oct 30 21:34:30.399808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.399764 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69d7877fb6-r25n9" podStartSLOduration=2.36635147 podStartE2EDuration="3.399753159s" totalImagesPullingTime="1.033401689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:30.399507673 +0000 UTC m=+332.595198793" watchObservedRunningTime="2026-10-30 21:34:30.399753159 +0000 UTC m=+332.595444277" Oct 30 21:34:30.414405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.414385 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b9864"] Oct 30 21:34:30.418651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.418631 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-b9864"] Oct 30 21:34:30.429296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.429276 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bjj87"] Oct 30 21:34:30.429440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.429421 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" podUID="329d08d1-37b0-4eaa-af23-9f670c3364ed" containerName="authorino" containerID="cri-o://e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab" gracePeriod=30 Oct 30 21:34:30.731099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.731076 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:34:30.783251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.783223 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc\" (UniqueName: \"kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc\") pod \"329d08d1-37b0-4eaa-af23-9f670c3364ed\" (UID: \"329d08d1-37b0-4eaa-af23-9f670c3364ed\") " Oct 30 21:34:30.785334 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.785310 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc" pod "329d08d1-37b0-4eaa-af23-9f670c3364ed" (UID: "329d08d1-37b0-4eaa-af23-9f670c3364ed"). InnerVolumeSpecName "kube-api-access-rmqbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:30.884145 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:30.884118 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqbc\" (UniqueName: \"kubernetes.io/projected/329d08d1-37b0-4eaa-af23-9f670c3364ed-kube-api-access-rmqbc\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:31.385008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.384977 6808 generic.go:356] "Generic (PLEG): container finished" podID="329d08d1-37b0-4eaa-af23-9f670c3364ed" containerID="e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab" exitCode=0 Oct 30 21:34:31.385426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.385072 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" event={"ID":"329d08d1-37b0-4eaa-af23-9f670c3364ed","Type":"ContainerDied","Data":"e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab"} Oct 30 21:34:31.385426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.385107 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" event={"ID":"329d08d1-37b0-4eaa-af23-9f670c3364ed","Type":"ContainerDied","Data":"76ef8853ef51dc5e2ef20a81bcc7a52bd58975f8cdf72d375402e6d6c33cdc8e"} Oct 30 21:34:31.385426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.385079 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bjj87" Oct 30 21:34:31.385426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.385123 6808 scope.go:122] "RemoveContainer" containerID="e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab" Oct 30 21:34:31.393695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.393680 6808 scope.go:122] "RemoveContainer" containerID="e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab" Oct 30 21:34:31.393931 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:34:31.393914 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab\": container with ID starting with e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab not found: ID does not exist" containerID="e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab" Oct 30 21:34:31.394005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.393939 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab"} err="failed to get container status \"e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab\": rpc error: code = NotFound desc = could not find container \"e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab\": container with ID starting with e4132330e3ebe29e8dc9e388577e90a47ab11c7ed927decc927ca935f82392ab not found: ID does not exist" Oct 30 21:34:31.418935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.418914 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bjj87"] Oct 30 21:34:31.426540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.426517 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5"] Oct 30 21:34:31.426964 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.426918 6808 cpu_manager.go:414] "containerMap: removing container" podUID="329d08d1-37b0-4eaa-af23-9f670c3364ed" containerName="authorino" Oct 30 21:34:31.427075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.426969 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329d08d1-37b0-4eaa-af23-9f670c3364ed" containerName="authorino" Oct 30 21:34:31.427075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.427014 6808 cpu_manager.go:414] "containerMap: removing container" podUID="c40e17b3-8c78-49f3-80cd-dc793dc80c91" containerName="authorino" Oct 30 21:34:31.427075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.427023 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40e17b3-8c78-49f3-80cd-dc793dc80c91" containerName="authorino" Oct 30 21:34:31.427228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.427114 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="329d08d1-37b0-4eaa-af23-9f670c3364ed" containerName="authorino" Oct 30 21:34:31.427228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.427131 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40e17b3-8c78-49f3-80cd-dc793dc80c91" containerName="authorino" Oct 30 21:34:31.430143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.430126 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bjj87"] Oct 30 21:34:31.430224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.430199 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.443125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.443094 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5"] Oct 30 21:34:31.488872 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.488848 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thv96\" (UniqueName: \"kubernetes.io/projected/e595af5c-5533-44ac-b2ac-87f30ea41af7-kube-api-access-thv96\") pod \"llmisvc-controller-manager-6466c554d8-2ckd5\" (UID: \"e595af5c-5533-44ac-b2ac-87f30ea41af7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.488998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.488889 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e595af5c-5533-44ac-b2ac-87f30ea41af7-cert\") pod \"llmisvc-controller-manager-6466c554d8-2ckd5\" (UID: \"e595af5c-5533-44ac-b2ac-87f30ea41af7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.589700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.589674 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e595af5c-5533-44ac-b2ac-87f30ea41af7-cert\") pod \"llmisvc-controller-manager-6466c554d8-2ckd5\" (UID: \"e595af5c-5533-44ac-b2ac-87f30ea41af7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.589790 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.589750 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thv96\" (UniqueName: \"kubernetes.io/projected/e595af5c-5533-44ac-b2ac-87f30ea41af7-kube-api-access-thv96\") pod \"llmisvc-controller-manager-6466c554d8-2ckd5\" (UID: \"e595af5c-5533-44ac-b2ac-87f30ea41af7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.592033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.592008 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e595af5c-5533-44ac-b2ac-87f30ea41af7-cert\") pod \"llmisvc-controller-manager-6466c554d8-2ckd5\" (UID: \"e595af5c-5533-44ac-b2ac-87f30ea41af7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.602274 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.602252 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thv96\" (UniqueName: \"kubernetes.io/projected/e595af5c-5533-44ac-b2ac-87f30ea41af7-kube-api-access-thv96\") pod \"llmisvc-controller-manager-6466c554d8-2ckd5\" (UID: \"e595af5c-5533-44ac-b2ac-87f30ea41af7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.741587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.741532 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:31.931443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.927653 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329d08d1-37b0-4eaa-af23-9f670c3364ed" path="/var/lib/kubelet/pods/329d08d1-37b0-4eaa-af23-9f670c3364ed/volumes" Oct 30 21:34:31.931443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:31.928212 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40e17b3-8c78-49f3-80cd-dc793dc80c91" path="/var/lib/kubelet/pods/c40e17b3-8c78-49f3-80cd-dc793dc80c91/volumes" Oct 30 21:34:32.013343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:32.013282 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5"] Oct 30 21:34:32.016320 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:34:32.016290 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode595af5c_5533_44ac_b2ac_87f30ea41af7.slice/crio-c0fa7bff1a9d86f9b6c3b34510fdcd5c589a3b1ab275aa8325fc4aec97eb8604 WatchSource:0}: Error finding container c0fa7bff1a9d86f9b6c3b34510fdcd5c589a3b1ab275aa8325fc4aec97eb8604: Status 404 returned error can't find the container with id c0fa7bff1a9d86f9b6c3b34510fdcd5c589a3b1ab275aa8325fc4aec97eb8604 Oct 30 21:34:32.391758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:32.391725 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" event={"ID":"e595af5c-5533-44ac-b2ac-87f30ea41af7","Type":"ContainerStarted","Data":"c0fa7bff1a9d86f9b6c3b34510fdcd5c589a3b1ab275aa8325fc4aec97eb8604"} Oct 30 21:34:33.398408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:33.398372 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" event={"ID":"e595af5c-5533-44ac-b2ac-87f30ea41af7","Type":"ContainerStarted","Data":"fffa720333d37c947406419c70419899b12f1ac4c26d39ba69b14bedcc5cdc39"} Oct 30 21:34:33.398873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:33.398480 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:34:33.419691 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:33.419649 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" podStartSLOduration=1.908300868 podStartE2EDuration="2.419637846s" totalImagesPullingTime="511.336978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:33.419132241 +0000 UTC m=+335.614823361" watchObservedRunningTime="2026-10-30 21:34:33.419637846 +0000 UTC m=+335.615328964" Oct 30 21:34:35.697130 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:34:35.697085 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:34:36.430707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.430674 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45"] Oct 30 21:34:36.430928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.430892 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" podUID="15df2642-616a-4b7a-a075-e451a5de50cf" containerName="maas-api" containerID="cri-o://5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0" gracePeriod=30 Oct 30 21:34:36.667169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.667147 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:36.729892 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.729817 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls\") pod \"15df2642-616a-4b7a-a075-e451a5de50cf\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " Oct 30 21:34:36.729892 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.729857 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh\" (UniqueName: \"kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh\") pod \"15df2642-616a-4b7a-a075-e451a5de50cf\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " Oct 30 21:34:36.730366 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.729938 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca\") pod \"15df2642-616a-4b7a-a075-e451a5de50cf\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " Oct 30 21:34:36.730366 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.730032 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs\") pod \"15df2642-616a-4b7a-a075-e451a5de50cf\" (UID: \"15df2642-616a-4b7a-a075-e451a5de50cf\") " Oct 30 21:34:36.730366 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.730322 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca" pod "15df2642-616a-4b7a-a075-e451a5de50cf" (UID: "15df2642-616a-4b7a-a075-e451a5de50cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:36.732110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.732076 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs" pod "15df2642-616a-4b7a-a075-e451a5de50cf" (UID: "15df2642-616a-4b7a-a075-e451a5de50cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:36.732110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.732095 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh" pod "15df2642-616a-4b7a-a075-e451a5de50cf" (UID: "15df2642-616a-4b7a-a075-e451a5de50cf"). InnerVolumeSpecName "kube-api-access-gkvxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:36.732246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.732146 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls" pod "15df2642-616a-4b7a-a075-e451a5de50cf" (UID: "15df2642-616a-4b7a-a075-e451a5de50cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:36.831428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.831399 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df2642-616a-4b7a-a075-e451a5de50cf-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:36.831428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.831426 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:36.831588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.831441 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df2642-616a-4b7a-a075-e451a5de50cf-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:36.831588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:36.831455 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkvxh\" (UniqueName: \"kubernetes.io/projected/15df2642-616a-4b7a-a075-e451a5de50cf-kube-api-access-gkvxh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:37.417442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.417413 6808 generic.go:356] "Generic (PLEG): container finished" podID="15df2642-616a-4b7a-a075-e451a5de50cf" containerID="5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0" exitCode=0 Oct 30 21:34:37.417584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.417457 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" Oct 30 21:34:37.417584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.417473 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" event={"ID":"15df2642-616a-4b7a-a075-e451a5de50cf","Type":"ContainerDied","Data":"5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0"} Oct 30 21:34:37.417584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.417519 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45" event={"ID":"15df2642-616a-4b7a-a075-e451a5de50cf","Type":"ContainerDied","Data":"a0b291f75a76dcedaac3a978bbe148adf53c34d6a8a7f98d13c553aed96b2c5b"} Oct 30 21:34:37.417584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.417539 6808 scope.go:122] "RemoveContainer" containerID="5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0" Oct 30 21:34:37.426599 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.426576 6808 scope.go:122] "RemoveContainer" containerID="5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0" Oct 30 21:34:37.426869 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:34:37.426841 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0\": container with ID starting with 5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0 not found: ID does not exist" containerID="5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0" Oct 30 21:34:37.426921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.426880 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0"} err="failed to get container status \"5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0\": rpc error: code = NotFound desc = could not find container \"5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0\": container with ID starting with 5ab34d9ae2602afa585c2ab3817212b342f084cb2efc766ea60294b6e6c848b0 not found: ID does not exist" Oct 30 21:34:37.442449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.442430 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45"] Oct 30 21:34:37.449712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.449689 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7mk45"] Oct 30 21:34:37.930735 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:34:37.930700 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15df2642-616a-4b7a-a075-e451a5de50cf" path="/var/lib/kubelet/pods/15df2642-616a-4b7a-a075-e451a5de50cf/volumes" Oct 30 21:35:04.404877 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.404844 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ckd5" Oct 30 21:35:04.452698 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.452664 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x"] Oct 30 21:35:04.452908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.452886 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" podUID="052edc80-445b-483d-b47e-ea9755040cfc" containerName="manager" containerID="cri-o://4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75" gracePeriod=30 Oct 30 21:35:04.697371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.697346 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:35:04.837774 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.837747 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert\" (UniqueName: \"kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert\") pod \"052edc80-445b-483d-b47e-ea9755040cfc\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " Oct 30 21:35:04.837937 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.837786 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt\" (UniqueName: \"kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt\") pod \"052edc80-445b-483d-b47e-ea9755040cfc\" (UID: \"052edc80-445b-483d-b47e-ea9755040cfc\") " Oct 30 21:35:04.839942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.839913 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert" pod "052edc80-445b-483d-b47e-ea9755040cfc" (UID: "052edc80-445b-483d-b47e-ea9755040cfc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:04.840050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.839970 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt" pod "052edc80-445b-483d-b47e-ea9755040cfc" (UID: "052edc80-445b-483d-b47e-ea9755040cfc"). InnerVolumeSpecName "kube-api-access-gljdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:04.939354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.939274 6808 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052edc80-445b-483d-b47e-ea9755040cfc-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:35:04.939354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:04.939306 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gljdt\" (UniqueName: \"kubernetes.io/projected/052edc80-445b-483d-b47e-ea9755040cfc-kube-api-access-gljdt\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:35:05.542414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.542379 6808 generic.go:356] "Generic (PLEG): container finished" podID="052edc80-445b-483d-b47e-ea9755040cfc" containerID="4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75" exitCode=0 Oct 30 21:35:05.542414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.542415 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" event={"ID":"052edc80-445b-483d-b47e-ea9755040cfc","Type":"ContainerDied","Data":"4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75"} Oct 30 21:35:05.542944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.542428 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" Oct 30 21:35:05.542944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.542457 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x" event={"ID":"052edc80-445b-483d-b47e-ea9755040cfc","Type":"ContainerDied","Data":"856723885a3213039ec1a56291732d4b6cf89a786d5c46b076a10e5bd6df90e5"} Oct 30 21:35:05.542944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.542481 6808 scope.go:122] "RemoveContainer" containerID="4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75" Oct 30 21:35:05.551199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.551182 6808 scope.go:122] "RemoveContainer" containerID="4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75" Oct 30 21:35:05.551471 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:35:05.551451 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75\": container with ID starting with 4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75 not found: ID does not exist" containerID="4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75" Oct 30 21:35:05.551516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.551479 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75"} err="failed to get container status \"4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75\": rpc error: code = NotFound desc = could not find container \"4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75\": container with ID starting with 4f04ed82949e83e03737444d920f92195619d6c747c8388fb5edda3204f58d75 not found: ID does not exist" Oct 30 21:35:05.615564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.615535 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x"] Oct 30 21:35:05.622101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.622075 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4js7x"] Oct 30 21:35:05.697096 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:35:05.697061 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:35:05.926368 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:05.926340 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052edc80-445b-483d-b47e-ea9755040cfc" path="/var/lib/kubelet/pods/052edc80-445b-483d-b47e-ea9755040cfc/volumes" Oct 30 21:35:35.699685 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:35:35.699644 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:35:48.603163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603128 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-794b4996f-5679g"] Oct 30 21:35:48.603698 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603668 6808 cpu_manager.go:414] "containerMap: removing container" podUID="15df2642-616a-4b7a-a075-e451a5de50cf" containerName="maas-api" Oct 30 21:35:48.603698 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603693 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15df2642-616a-4b7a-a075-e451a5de50cf" containerName="maas-api" Oct 30 21:35:48.603853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603719 6808 cpu_manager.go:414] "containerMap: removing container" podUID="052edc80-445b-483d-b47e-ea9755040cfc" containerName="manager" Oct 30 21:35:48.603853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603728 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052edc80-445b-483d-b47e-ea9755040cfc" containerName="manager" Oct 30 21:35:48.603853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603816 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="052edc80-445b-483d-b47e-ea9755040cfc" containerName="manager" Oct 30 21:35:48.603853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.603837 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="15df2642-616a-4b7a-a075-e451a5de50cf" containerName="maas-api" Oct 30 21:35:48.606915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.606898 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.618735 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.618706 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-794b4996f-5679g"] Oct 30 21:35:48.725670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.725637 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d-tls-cert\") pod \"authorino-794b4996f-5679g\" (UID: \"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d\") " pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.725670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.725668 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99fd\" (UniqueName: \"kubernetes.io/projected/c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d-kube-api-access-d99fd\") pod \"authorino-794b4996f-5679g\" (UID: \"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d\") " pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.826585 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.826552 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d-tls-cert\") pod \"authorino-794b4996f-5679g\" (UID: \"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d\") " pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.826744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.826596 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99fd\" (UniqueName: \"kubernetes.io/projected/c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d-kube-api-access-d99fd\") pod \"authorino-794b4996f-5679g\" (UID: \"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d\") " pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.829078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.829052 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d-tls-cert\") pod \"authorino-794b4996f-5679g\" (UID: \"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d\") " pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.839021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.838993 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99fd\" (UniqueName: \"kubernetes.io/projected/c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d-kube-api-access-d99fd\") pod \"authorino-794b4996f-5679g\" (UID: \"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d\") " pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:48.918779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:48.918751 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-794b4996f-5679g" Oct 30 21:35:49.190155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:49.190080 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-794b4996f-5679g"] Oct 30 21:35:49.196017 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:35:49.195991 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc630ee4a_57f1_4e4d_b293_0bd0ad06cb0d.slice/crio-3814bf607654c0f5c75aa25c143129da082893b583abf96281dd65f669414814 WatchSource:0}: Error finding container 3814bf607654c0f5c75aa25c143129da082893b583abf96281dd65f669414814: Status 404 returned error can't find the container with id 3814bf607654c0f5c75aa25c143129da082893b583abf96281dd65f669414814 Oct 30 21:35:49.752536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:49.752497 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-794b4996f-5679g" event={"ID":"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d","Type":"ContainerStarted","Data":"3814bf607654c0f5c75aa25c143129da082893b583abf96281dd65f669414814"} Oct 30 21:35:51.764662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:51.764626 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-794b4996f-5679g" event={"ID":"c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d","Type":"ContainerStarted","Data":"d5f1c11198018af8ded7fe5de7f714c57c7acc9555daf7003fd89498f350b394"} Oct 30 21:35:51.788315 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:51.788249 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-794b4996f-5679g" podStartSLOduration=2.2161400589999998 podStartE2EDuration="3.788229923s" totalImagesPullingTime="1.572089864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:51.786858138 +0000 UTC m=+413.982549258" watchObservedRunningTime="2026-10-30 21:35:51.788229923 +0000 UTC m=+413.983921039" Oct 30 21:35:51.820233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:51.820195 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69d7877fb6-r25n9"] Oct 30 21:35:51.820478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:51.820432 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-69d7877fb6-r25n9" podUID="2056af91-4fda-4711-b425-d85693ee2eac" containerName="authorino" containerID="cri-o://757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8" gracePeriod=30 Oct 30 21:35:52.083149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.083118 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:35:52.163128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.163085 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert\" (UniqueName: \"kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert\") pod \"2056af91-4fda-4711-b425-d85693ee2eac\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " Oct 30 21:35:52.163370 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.163141 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq\" (UniqueName: \"kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq\") pod \"2056af91-4fda-4711-b425-d85693ee2eac\" (UID: \"2056af91-4fda-4711-b425-d85693ee2eac\") " Oct 30 21:35:52.165770 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.165740 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq" pod "2056af91-4fda-4711-b425-d85693ee2eac" (UID: "2056af91-4fda-4711-b425-d85693ee2eac"). InnerVolumeSpecName "kube-api-access-f5cvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:52.175064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.175031 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert" pod "2056af91-4fda-4711-b425-d85693ee2eac" (UID: "2056af91-4fda-4711-b425-d85693ee2eac"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:52.263977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.263930 6808 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2056af91-4fda-4711-b425-d85693ee2eac-tls-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:35:52.263977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.263971 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5cvq\" (UniqueName: \"kubernetes.io/projected/2056af91-4fda-4711-b425-d85693ee2eac-kube-api-access-f5cvq\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:35:52.770665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.770629 6808 generic.go:356] "Generic (PLEG): container finished" podID="2056af91-4fda-4711-b425-d85693ee2eac" containerID="757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8" exitCode=0 Oct 30 21:35:52.771174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.770684 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69d7877fb6-r25n9" Oct 30 21:35:52.771174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.770714 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69d7877fb6-r25n9" event={"ID":"2056af91-4fda-4711-b425-d85693ee2eac","Type":"ContainerDied","Data":"757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8"} Oct 30 21:35:52.771174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.770759 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69d7877fb6-r25n9" event={"ID":"2056af91-4fda-4711-b425-d85693ee2eac","Type":"ContainerDied","Data":"e6e27dff4ec551b61b04103a62f7e1e2cc72b66b99d3479b1fdfa9e9d509107e"} Oct 30 21:35:52.771174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.770779 6808 scope.go:122] "RemoveContainer" containerID="757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8" Oct 30 21:35:52.780354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.780332 6808 scope.go:122] "RemoveContainer" containerID="757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8" Oct 30 21:35:52.788350 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:35:52.786758 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8\": container with ID starting with 757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8 not found: ID does not exist" containerID="757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8" Oct 30 21:35:52.788350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.786826 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8"} err="failed to get container status \"757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8\": rpc error: code = NotFound desc = could not find container \"757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8\": container with ID starting with 757ac1562f9f51ef2f87578dffa9451c7574b8a268622a178cfda7ec49a8fca8 not found: ID does not exist" Oct 30 21:35:52.808037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.808007 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69d7877fb6-r25n9"] Oct 30 21:35:52.815704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:52.815674 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69d7877fb6-r25n9"] Oct 30 21:35:53.929527 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:35:53.929466 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2056af91-4fda-4711-b425-d85693ee2eac" path="/var/lib/kubelet/pods/2056af91-4fda-4711-b425-d85693ee2eac/volumes" Oct 30 21:36:05.697567 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:36:05.697526 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:36:07.013015 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.012981 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl"] Oct 30 21:36:07.013392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.013328 6808 cpu_manager.go:414] "containerMap: removing container" podUID="2056af91-4fda-4711-b425-d85693ee2eac" containerName="authorino" Oct 30 21:36:07.013392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.013340 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2056af91-4fda-4711-b425-d85693ee2eac" containerName="authorino" Oct 30 21:36:07.013468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.013404 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="2056af91-4fda-4711-b425-d85693ee2eac" containerName="authorino" Oct 30 21:36:07.017636 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.017618 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.041514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.041491 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl"] Oct 30 21:36:07.102442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.102421 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wb984\"" Oct 30 21:36:07.198588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198560 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198601 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198617 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198705 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198732 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvvz\" (UniqueName: \"kubernetes.io/projected/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-kube-api-access-4pvvz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198751 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198794 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198871 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.198934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.198898 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299597 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299626 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299647 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299693 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299743 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299768 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.299828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299803 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.299832 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvvz\" (UniqueName: \"kubernetes.io/projected/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-kube-api-access-4pvvz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.300011 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.300048 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.300065 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.300130 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.300157 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.300623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.300605 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.302016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.301991 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.302426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.302407 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.314462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.314440 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.320519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.320500 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvvz\" (UniqueName: \"kubernetes.io/projected/55d693b2-3c4e-48e2-977f-d4cbef44f3b3-kube-api-access-4pvvz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w8ptl\" (UID: \"55d693b2-3c4e-48e2-977f-d4cbef44f3b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.332321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.332305 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:07.614345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.614278 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl"] Oct 30 21:36:07.617646 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:36:07.617619 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d693b2_3c4e_48e2_977f_d4cbef44f3b3.slice/crio-5077620c295aa7bf23318d301f95a0b081441a3361116c1c52356038707b7def WatchSource:0}: Error finding container 5077620c295aa7bf23318d301f95a0b081441a3361116c1c52356038707b7def: Status 404 returned error can't find the container with id 5077620c295aa7bf23318d301f95a0b081441a3361116c1c52356038707b7def Oct 30 21:36:07.619514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.619499 6808 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:36:07.837841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:07.837813 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" event={"ID":"55d693b2-3c4e-48e2-977f-d4cbef44f3b3","Type":"ContainerStarted","Data":"5077620c295aa7bf23318d301f95a0b081441a3361116c1c52356038707b7def"} Oct 30 21:36:10.645471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:10.645443 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:10.645769 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:10.645507 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:10.645769 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:10.645535 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:10.853051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:10.853019 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" event={"ID":"55d693b2-3c4e-48e2-977f-d4cbef44f3b3","Type":"ContainerStarted","Data":"7fdff1ff5fb878786fb8e554742a722eebb27e66f2425bae8619c4544a88e3fa"} Oct 30 21:36:10.880356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:10.880297 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podStartSLOduration=1.855260498 podStartE2EDuration="4.880194017s" totalImagesPullingTime="3.024933519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:10.877869681 +0000 UTC m=+433.073560801" watchObservedRunningTime="2026-10-30 21:36:10.880194017 +0000 UTC m=+433.075885130" Oct 30 21:36:11.333323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:11.333284 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:11.334867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:11.334840 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:11.335025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:11.334896 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:12.332739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:12.332703 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:12.333137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:12.332757 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:13.333619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:13.333577 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:13.334058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:13.333646 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:14.333224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:14.333185 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:14.333393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:14.333248 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:15.333094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:15.333063 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:15.333510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:15.333113 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:16.333140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:16.333107 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:16.333536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:16.333170 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:17.333304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:17.333272 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:36:17.333726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:17.333335 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:36:18.730237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:18.730203 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:18.731318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:18.731295 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:18.731917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:18.731901 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" Oct 30 21:36:21.925733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.925704 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm"] Oct 30 21:36:21.929341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.929312 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939263 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939296 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939325 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5j64\" (UniqueName: \"kubernetes.io/projected/ce40587d-41ac-4215-9c6c-f0298ebec6a6-kube-api-access-w5j64\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939398 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939431 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939461 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939478 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939577 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.939641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.939615 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:21.945435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:21.945411 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm"] Oct 30 21:36:22.040206 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040170 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040222 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040250 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040289 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040334 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040543 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040370 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040543 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040401 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5j64\" (UniqueName: \"kubernetes.io/projected/ce40587d-41ac-4215-9c6c-f0298ebec6a6-kube-api-access-w5j64\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040543 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040428 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040543 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040479 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040672 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040726 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040784 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.040876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.040822 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.041233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.041211 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.042387 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.042365 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.042791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.042772 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.055000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.054975 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce40587d-41ac-4215-9c6c-f0298ebec6a6-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.055480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.055411 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5j64\" (UniqueName: \"kubernetes.io/projected/ce40587d-41ac-4215-9c6c-f0298ebec6a6-kube-api-access-w5j64\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hxfjm\" (UID: \"ce40587d-41ac-4215-9c6c-f0298ebec6a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.245574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.245508 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:22.523611 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.523539 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm"] Oct 30 21:36:22.527130 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:36:22.527102 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce40587d_41ac_4215_9c6c_f0298ebec6a6.slice/crio-7f41b9df075b322febeaa121ed1f52266b44114250bf54bd89f3f665442ca1d4 WatchSource:0}: Error finding container 7f41b9df075b322febeaa121ed1f52266b44114250bf54bd89f3f665442ca1d4: Status 404 returned error can't find the container with id 7f41b9df075b322febeaa121ed1f52266b44114250bf54bd89f3f665442ca1d4 Oct 30 21:36:22.530192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.530163 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:22.530249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.530225 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:22.530284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.530257 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:22.922672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.921242 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" event={"ID":"ce40587d-41ac-4215-9c6c-f0298ebec6a6","Type":"ContainerStarted","Data":"080e6cbe4811032527ff8dcb70a8a25b115869e4b482c8a9eab9fc3c4c6354d7"} Oct 30 21:36:22.922672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.921318 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" event={"ID":"ce40587d-41ac-4215-9c6c-f0298ebec6a6","Type":"ContainerStarted","Data":"7f41b9df075b322febeaa121ed1f52266b44114250bf54bd89f3f665442ca1d4"} Oct 30 21:36:22.956973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:22.956721 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podStartSLOduration=1.95670083 podStartE2EDuration="1.95670083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:22.952218585 +0000 UTC m=+445.147909708" watchObservedRunningTime="2026-10-30 21:36:22.95670083 +0000 UTC m=+445.152391948" Oct 30 21:36:23.253030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:23.252304 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:23.264986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:23.262079 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:23.264986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:23.262152 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:24.251803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:24.251330 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:24.251803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:24.251397 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:25.265076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:25.265037 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:25.265634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:25.265097 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:26.260973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:26.254358 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:26.260973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:26.254424 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:27.255973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:27.247351 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:27.255973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:27.247422 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:28.265327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:28.265289 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:28.265736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:28.265350 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:29.246494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:29.246452 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:36:29.246679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:29.246534 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:36:30.623300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:30.623265 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:30.623792 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:30.623580 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:30.624519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:30.624497 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" Oct 30 21:36:35.696257 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:36:35.696222 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:36:36.941811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.941774 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn"] Oct 30 21:36:36.947457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.947427 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.959614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.959586 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn"] Oct 30 21:36:36.984474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984447 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984529 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984558 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984592 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984718 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984756 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswq7\" (UniqueName: \"kubernetes.io/projected/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-kube-api-access-vswq7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984775 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984796 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:36.984861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:36.984815 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085487 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085450 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085500 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswq7\" (UniqueName: \"kubernetes.io/projected/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-kube-api-access-vswq7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085527 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085549 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085576 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085605 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085676 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085705 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085749 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.085934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.085888 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.086175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.086118 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.086175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.086137 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.086262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.086184 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.086576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.086508 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.087876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.087857 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.088103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.088086 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.098498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.098473 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.098708 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.098692 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswq7\" (UniqueName: \"kubernetes.io/projected/c3699ff6-469e-42ec-9f4d-c9e124f1f3cc-kube-api-access-vswq7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wtthn\" (UID: \"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.266281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.266210 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:37.553005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.552855 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn"] Oct 30 21:36:37.556247 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:36:37.556216 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3699ff6_469e_42ec_9f4d_c9e124f1f3cc.slice/crio-da182bc494d8d30eb2ce9d33500be6d1677d898e7d6c62dfd0e3b55c381f1f2e WatchSource:0}: Error finding container da182bc494d8d30eb2ce9d33500be6d1677d898e7d6c62dfd0e3b55c381f1f2e: Status 404 returned error can't find the container with id da182bc494d8d30eb2ce9d33500be6d1677d898e7d6c62dfd0e3b55c381f1f2e Oct 30 21:36:37.558613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.558577 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:37.558725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.558657 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:37.558725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:37.558700 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:38.004647 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:38.004612 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" event={"ID":"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc","Type":"ContainerStarted","Data":"33e3e0340866332d7ed5491f9af55d3dc57479429434ded358487aa714986703"} Oct 30 21:36:38.005071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:38.004665 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" event={"ID":"c3699ff6-469e-42ec-9f4d-c9e124f1f3cc","Type":"ContainerStarted","Data":"da182bc494d8d30eb2ce9d33500be6d1677d898e7d6c62dfd0e3b55c381f1f2e"} Oct 30 21:36:38.031722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:38.031674 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podStartSLOduration=2.031657247 podStartE2EDuration="2.031657247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:38.029341581 +0000 UTC m=+460.225032701" watchObservedRunningTime="2026-10-30 21:36:38.031657247 +0000 UTC m=+460.227348366" Oct 30 21:36:38.266733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:38.266632 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:38.268171 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:38.268144 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:38.268267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:38.268194 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:39.267490 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:39.267452 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:39.267977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:39.267504 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:40.266638 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:40.266555 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:40.266638 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:40.266623 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:41.267548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:41.267507 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:41.267972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:41.267579 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:42.267502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:42.267466 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:42.267709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:42.267524 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:43.266973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:43.266920 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:43.267148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:43.267013 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:44.267470 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:44.267436 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:44.267859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:44.267499 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:36:45.707026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:45.706995 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:45.707509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:45.707299 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:45.708284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:45.708265 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" Oct 30 21:36:53.248728 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.248688 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9"] Oct 30 21:36:53.258149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.258125 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.262105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.262070 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9"] Oct 30 21:36:53.267682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.267661 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1d8034-openshift-default-dockercfg-hl29t\"" Oct 30 21:36:53.356668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356629 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.356835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356709 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.356835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356734 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.356835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356780 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl89x\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.357096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356850 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.357096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356888 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.357096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356913 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.357096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.356937 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.357096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.357004 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.457972 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458059 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458084 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458134 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl89x\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458212 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458250 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458275 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458299 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458314 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458337 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458748 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.458963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.458790 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.459072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.459048 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.459857 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.459832 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.461576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.461552 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.462136 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.462120 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.496574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.496542 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl89x\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.496824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.496802 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.502456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.502396 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv"] Oct 30 21:36:53.507793 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.507774 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.544321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.544297 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv"] Oct 30 21:36:53.558178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.558156 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-db0b54-openshift-default-dockercfg-ltx2s\"" Oct 30 21:36:53.579487 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.579441 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660184 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660237 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660268 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660314 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660400 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660435 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgk4x\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660489 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660515 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.660764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.660542 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762212 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762128 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762212 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762173 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgk4x\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762225 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762249 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762275 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762334 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762366 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762389 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762436 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762691 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.762802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.762766 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.763318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.763030 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.763318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.763262 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.764037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.763877 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.764873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.764848 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.766221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.766195 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.775412 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.775392 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.775726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.775709 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgk4x\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.825977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.821130 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:53.880724 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.880294 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9"] Oct 30 21:36:53.881039 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:36:53.880990 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f3f5a4_1224_4297_91da_291efc553540.slice/crio-a7f4381f4ea03cfd0a57a6128d76fbde388906f4dce31f7df506fd55f37a3755 WatchSource:0}: Error finding container a7f4381f4ea03cfd0a57a6128d76fbde388906f4dce31f7df506fd55f37a3755: Status 404 returned error can't find the container with id a7f4381f4ea03cfd0a57a6128d76fbde388906f4dce31f7df506fd55f37a3755 Oct 30 21:36:53.886443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.885476 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:53.886443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.885564 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:53.886443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:53.885606 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:54.096705 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.096608 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" event={"ID":"d2f3f5a4-1224-4297-91da-291efc553540","Type":"ContainerStarted","Data":"a05f08f8ff52cef1fb53c70356968f8f09272497fa00a352fafc4fd8a03f321c"} Oct 30 21:36:54.096705 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.096681 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" event={"ID":"d2f3f5a4-1224-4297-91da-291efc553540","Type":"ContainerStarted","Data":"a7f4381f4ea03cfd0a57a6128d76fbde388906f4dce31f7df506fd55f37a3755"} Oct 30 21:36:54.127091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.127033 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podStartSLOduration=1.127020924 podStartE2EDuration="1.127020924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:54.126860261 +0000 UTC m=+476.322551392" watchObservedRunningTime="2026-10-30 21:36:54.127020924 +0000 UTC m=+476.322712044" Oct 30 21:36:54.138356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.138221 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv"] Oct 30 21:36:54.143503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.143467 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:54.143631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.143552 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:54.143631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.143580 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:36:54.580225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.580187 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:54.585542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:54.585488 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:55.102696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.102652 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" event={"ID":"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32","Type":"ContainerStarted","Data":"3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3"} Oct 30 21:36:55.102871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.102716 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" event={"ID":"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32","Type":"ContainerStarted","Data":"0bc3ab4c004a2168c4e364100372ae234c50a3af594af74d294081b3e0a82ac3"} Oct 30 21:36:55.103104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.103084 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:55.104115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.104092 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:36:55.134688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.132512 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podStartSLOduration=2.132492327 podStartE2EDuration="2.132492327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:55.125872701 +0000 UTC m=+477.321563831" watchObservedRunningTime="2026-10-30 21:36:55.132492327 +0000 UTC m=+477.328183450" Oct 30 21:36:55.822052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.822019 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:55.827768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:55.827738 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:56.107992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:56.107882 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:36:56.108724 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:36:56.108695 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:37:01.859420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.859382 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv"] Oct 30 21:37:01.864175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.864150 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:01.872068 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.872048 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1a81ae-dockercfg-tcrfr\"" Oct 30 21:37:01.876150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.876128 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1a81ae\"" Oct 30 21:37:01.878198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.878176 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv"] Oct 30 21:37:01.895528 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.895492 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:01.895734 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.895709 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28tm\" (UniqueName: \"kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm\") pod \"payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:02.001993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.999752 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28tm\" (UniqueName: \"kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm\") pod \"payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:02.001993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:01.999935 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:02.001993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:02.001099 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:02.026787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:02.026756 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28tm\" (UniqueName: \"kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm\") pod \"payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:02.184971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:02.179152 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:02.563222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:02.563195 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv"] Oct 30 21:37:02.566079 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:02.566049 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1458c21_ef04_42e2_a973_dff27829548f.slice/crio-b0726fcc0dcb6ab081c217723c901a943ccb11eebcb0df07074db3bc0dc08eeb WatchSource:0}: Error finding container b0726fcc0dcb6ab081c217723c901a943ccb11eebcb0df07074db3bc0dc08eeb: Status 404 returned error can't find the container with id b0726fcc0dcb6ab081c217723c901a943ccb11eebcb0df07074db3bc0dc08eeb Oct 30 21:37:03.035240 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.034847 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2"] Oct 30 21:37:03.041882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.041856 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.070534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.070504 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2"] Oct 30 21:37:03.111727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.111693 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.111898 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.111784 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvlh\" (UniqueName: \"kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh\") pod \"payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.120554 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.120519 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-db0b54\"" Oct 30 21:37:03.136775 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.136742 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-db0b54-dockercfg-p4k8x\"" Oct 30 21:37:03.177523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.177441 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" event={"ID":"d1458c21-ef04-42e2-a973-dff27829548f","Type":"ContainerStarted","Data":"b0726fcc0dcb6ab081c217723c901a943ccb11eebcb0df07074db3bc0dc08eeb"} Oct 30 21:37:03.220971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.217484 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvlh\" (UniqueName: \"kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh\") pod \"payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.220971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.217647 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.220971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.218619 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.235583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.235553 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvlh\" (UniqueName: \"kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh\") pod \"payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.365007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.364923 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:03.648404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:03.648339 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2"] Oct 30 21:37:03.650832 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:03.650788 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8987be6_af66_4ccd_892b_6cb2c7c448fc.slice/crio-7e395325a16efb2e729a1b74286056596a501f2460128e302a96d784c8d8a189 WatchSource:0}: Error finding container 7e395325a16efb2e729a1b74286056596a501f2460128e302a96d784c8d8a189: Status 404 returned error can't find the container with id 7e395325a16efb2e729a1b74286056596a501f2460128e302a96d784c8d8a189 Oct 30 21:37:04.186606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.186522 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" event={"ID":"f8987be6-af66-4ccd-892b-6cb2c7c448fc","Type":"ContainerStarted","Data":"7e395325a16efb2e729a1b74286056596a501f2460128e302a96d784c8d8a189"} Oct 30 21:37:04.904099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.904064 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc"] Oct 30 21:37:04.908466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.908443 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:04.914984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.914934 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:37:04.918779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.918729 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-1a81ae-serving-cert\"" Oct 30 21:37:04.918913 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.918844 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4ttw5\"" Oct 30 21:37:04.922389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.922363 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc"] Oct 30 21:37:04.929702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.929042 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:37:04.931489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:04.930042 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:37:05.044721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.037450 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.044721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.037500 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.044721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.037549 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkvg9\" (UniqueName: \"kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.044721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.037606 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.138529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.138503 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.138664 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.138537 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.138664 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.138564 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvg9\" (UniqueName: \"kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.138664 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.138601 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.139197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.139177 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.140928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.140912 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.141076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.141051 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.151368 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.151333 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkvg9\" (UniqueName: \"kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9\") pod \"maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.194010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.193981 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" event={"ID":"d1458c21-ef04-42e2-a973-dff27829548f","Type":"ContainerStarted","Data":"9dd3947b1c865d2701183ef84ffcc564521fb869468d0ccf552db9a83a68fbd1"} Oct 30 21:37:05.194414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.194164 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:05.195518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.195494 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" event={"ID":"f8987be6-af66-4ccd-892b-6cb2c7c448fc","Type":"ContainerStarted","Data":"c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a"} Oct 30 21:37:05.195628 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.195604 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:05.214615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.214570 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" podStartSLOduration=1.7984380930000001 podStartE2EDuration="4.214556807s" totalImagesPullingTime="2.416118714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:05.212539035 +0000 UTC m=+487.408230156" watchObservedRunningTime="2026-10-30 21:37:05.214556807 +0000 UTC m=+487.410247926" Oct 30 21:37:05.230546 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.230522 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:05.232442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.232403 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" podStartSLOduration=0.809204659 podStartE2EDuration="2.23239144s" totalImagesPullingTime="1.423186781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:05.232005704 +0000 UTC m=+487.427696820" watchObservedRunningTime="2026-10-30 21:37:05.23239144 +0000 UTC m=+487.428082558" Oct 30 21:37:05.517431 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.517259 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc"] Oct 30 21:37:05.520125 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:05.520099 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb796528b_ee5b_4b00_8fbd_7f4983eab248.slice/crio-f15b8f33b4258b90c94b53310160d72cbc97bc7b50de1aa10d1c23e7878d2dfa WatchSource:0}: Error finding container f15b8f33b4258b90c94b53310160d72cbc97bc7b50de1aa10d1c23e7878d2dfa: Status 404 returned error can't find the container with id f15b8f33b4258b90c94b53310160d72cbc97bc7b50de1aa10d1c23e7878d2dfa Oct 30 21:37:05.723098 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:37:05.721728 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:37:05.770554 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.770478 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw"] Oct 30 21:37:05.776009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.775982 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.780682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.780648 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-caa3dc-serving-cert\"" Oct 30 21:37:05.786345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.786279 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw"] Oct 30 21:37:05.844834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.844810 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.845005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.844843 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.845005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.844864 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.845005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.844892 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8k9\" (UniqueName: \"kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.945759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.945732 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.945867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.945784 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.945867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.945818 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.945867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.945850 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8k9\" (UniqueName: \"kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.946661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.946633 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.948326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.948299 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.948416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.948298 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:05.958189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:05.958165 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8k9\" (UniqueName: \"kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9\") pod \"maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:06.095391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:06.095305 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:06.280777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:06.279297 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" event={"ID":"b796528b-ee5b-4b00-8fbd-7f4983eab248","Type":"ContainerStarted","Data":"fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7"} Oct 30 21:37:06.280777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:06.279355 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" event={"ID":"b796528b-ee5b-4b00-8fbd-7f4983eab248","Type":"ContainerStarted","Data":"f15b8f33b4258b90c94b53310160d72cbc97bc7b50de1aa10d1c23e7878d2dfa"} Oct 30 21:37:06.280777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:06.279862 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:06.306318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:06.306254 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" podStartSLOduration=1.898622315 podStartE2EDuration="2.306234795s" totalImagesPullingTime="407.61248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:06.304501717 +0000 UTC m=+488.500192850" watchObservedRunningTime="2026-10-30 21:37:06.306234795 +0000 UTC m=+488.501925915" Oct 30 21:37:06.389652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:06.389625 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw"] Oct 30 21:37:06.398986 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:06.393073 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b9d2e9_be36_448d_9ad7_cbeaab68a5c1.slice/crio-6453b500271cd75f5edf5f0646f685489578f991cf19c37eeba266e327fa7144 WatchSource:0}: Error finding container 6453b500271cd75f5edf5f0646f685489578f991cf19c37eeba266e327fa7144: Status 404 returned error can't find the container with id 6453b500271cd75f5edf5f0646f685489578f991cf19c37eeba266e327fa7144 Oct 30 21:37:07.285195 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.285155 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" event={"ID":"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1","Type":"ContainerStarted","Data":"12aecbc86354713528e9d1e73bdcb7fd4d83d3a2846be208e0ca947fee298ca2"} Oct 30 21:37:07.285750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.285235 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" event={"ID":"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1","Type":"ContainerStarted","Data":"6453b500271cd75f5edf5f0646f685489578f991cf19c37eeba266e327fa7144"} Oct 30 21:37:07.285750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.285431 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:07.307271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.307214 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" podStartSLOduration=1.8802818970000001 podStartE2EDuration="2.307198664s" totalImagesPullingTime="426.916767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:07.306407588 +0000 UTC m=+489.502098707" watchObservedRunningTime="2026-10-30 21:37:07.307198664 +0000 UTC m=+489.502889784" Oct 30 21:37:07.600025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.599921 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp"] Oct 30 21:37:07.604123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.604100 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.608139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.608117 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-db0b54-serving-cert\"" Oct 30 21:37:07.615222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.614897 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp"] Oct 30 21:37:07.678455 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.678424 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.678585 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.678500 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8c7\" (UniqueName: \"kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.678585 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.678569 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.678697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.678613 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.779472 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.779432 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8c7\" (UniqueName: \"kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.779472 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.779475 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.779721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.779515 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.779721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.779616 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.780392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.780363 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.782002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.781974 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.782111 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.782093 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.791399 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.791375 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8c7\" (UniqueName: \"kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7\") pod \"maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:07.921476 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:07.921450 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:08.199877 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:08.199849 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp"] Oct 30 21:37:08.201750 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:08.201694 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5261f014_d469_4320_b521_a15cbc87ce02.slice/crio-6401e9af0396aff5eeb14b237a7618a1c6c0c10152f5d16f088a929626683751 WatchSource:0}: Error finding container 6401e9af0396aff5eeb14b237a7618a1c6c0c10152f5d16f088a929626683751: Status 404 returned error can't find the container with id 6401e9af0396aff5eeb14b237a7618a1c6c0c10152f5d16f088a929626683751 Oct 30 21:37:08.296980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:08.296764 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" event={"ID":"5261f014-d469-4320-b521-a15cbc87ce02","Type":"ContainerStarted","Data":"6401e9af0396aff5eeb14b237a7618a1c6c0c10152f5d16f088a929626683751"} Oct 30 21:37:09.302276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:09.302234 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" event={"ID":"5261f014-d469-4320-b521-a15cbc87ce02","Type":"ContainerStarted","Data":"e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7"} Oct 30 21:37:09.302758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:09.302317 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:09.326971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:09.326883 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" podStartSLOduration=1.92171146 podStartE2EDuration="2.326864887s" totalImagesPullingTime="405.153427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:09.325286365 +0000 UTC m=+491.520977501" watchObservedRunningTime="2026-10-30 21:37:09.326864887 +0000 UTC m=+491.522556007" Oct 30 21:37:13.304188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:13.304135 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:37:13.304687 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:13.304648 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:37:15.330202 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:15.330174 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:37:16.291142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:16.291112 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:37:16.301451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:16.301424 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:37:20.984696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:20.984663 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd"] Oct 30 21:37:20.987109 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:20.987092 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.012110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.012087 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"default-dockercfg-fm9ng\"" Oct 30 21:37:21.012110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.012099 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"simulator-w0-000f64-kserve-self-signed-certs\"" Oct 30 21:37:21.017834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.017809 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd"] Oct 30 21:37:21.018991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.018932 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"openshift-service-ca.crt\"" Oct 30 21:37:21.019115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.019018 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"kube-root-ca.crt\"" Oct 30 21:37:21.131671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.131631 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.131863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.131683 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pl67\" (UniqueName: \"kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.131863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.131753 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.131863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.131803 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.131863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.131836 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.132100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.131887 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233119 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233173 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pl67\" (UniqueName: \"kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233205 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233242 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233276 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233305 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233609 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233663 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.233761 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.233707 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.235684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.235653 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.235844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.235825 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.246106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.246086 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pl67\" (UniqueName: \"kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67\") pod \"simulator-w0-000f64-kserve-8695898b47-rrqmd\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.298084 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.298059 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:21.444706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.444673 6808 generic.go:356] "Generic (PLEG): container finished" podID="d2f3f5a4-1224-4297-91da-291efc553540" containerID="a05f08f8ff52cef1fb53c70356968f8f09272497fa00a352fafc4fd8a03f321c" exitCode=0 Oct 30 21:37:21.444893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.444757 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" event={"ID":"d2f3f5a4-1224-4297-91da-291efc553540","Type":"ContainerDied","Data":"a05f08f8ff52cef1fb53c70356968f8f09272497fa00a352fafc4fd8a03f321c"} Oct 30 21:37:21.445222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.445203 6808 scope.go:122] "RemoveContainer" containerID="a05f08f8ff52cef1fb53c70356968f8f09272497fa00a352fafc4fd8a03f321c" Oct 30 21:37:21.446536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.446510 6808 generic.go:356] "Generic (PLEG): container finished" podID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerID="3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3" exitCode=0 Oct 30 21:37:21.446634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.446585 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" event={"ID":"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32","Type":"ContainerDied","Data":"3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3"} Oct 30 21:37:21.446749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.446724 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:21.446832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.446792 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:21.446875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.446829 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:21.447006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.446989 6808 scope.go:122] "RemoveContainer" containerID="3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3" Oct 30 21:37:21.449149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.449085 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:21.449149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.449142 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:21.449259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.449172 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:37:21.608901 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:21.607707 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd"] Oct 30 21:37:21.612477 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:21.612439 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48ec31e2_b484_45a8_9888_35c505e1752b.slice/crio-bea616915ea8904c9c64832d6a95b9e0adea077798459959e97e7dcb36e26b14 WatchSource:0}: Error finding container bea616915ea8904c9c64832d6a95b9e0adea077798459959e97e7dcb36e26b14: Status 404 returned error can't find the container with id bea616915ea8904c9c64832d6a95b9e0adea077798459959e97e7dcb36e26b14 Oct 30 21:37:22.569970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:22.569265 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" event={"ID":"48ec31e2-b484-45a8-9888-35c505e1752b","Type":"ContainerStarted","Data":"bea616915ea8904c9c64832d6a95b9e0adea077798459959e97e7dcb36e26b14"} Oct 30 21:37:22.588025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:22.587005 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" event={"ID":"d2f3f5a4-1224-4297-91da-291efc553540","Type":"ContainerStarted","Data":"dd3e450d0fe939faa6663a1e51bfdf745b6e2b45e9fbb63c78f8aa19d7ab5787"} Oct 30 21:37:22.591600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:22.590930 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" event={"ID":"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32","Type":"ContainerStarted","Data":"55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239"} Oct 30 21:37:22.829024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:22.826451 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:37:22.833846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:22.833810 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:37:22.834027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:22.833875 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:37:23.587972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:23.587846 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:37:23.588497 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:23.587998 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:37:23.588497 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:23.588065 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:37:23.822756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:23.822300 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:37:23.822756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:23.822376 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:37:24.586840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:24.586362 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:37:24.586840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:24.586428 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:37:24.824848 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:24.824384 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:37:24.824848 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:24.824463 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:37:25.115973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:25.115642 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:37:25.583914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:25.583413 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.583914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:25.583478 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:37:25.825238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:25.825195 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.825697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:25.825266 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:37:26.108725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:26.108682 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:37:26.590973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:26.590368 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:37:26.590973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:26.590435 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:37:26.821690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:26.821661 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:37:26.821804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:26.821721 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:37:27.580355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:27.580208 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:37:27.580355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:27.580269 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:37:27.680198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:27.680155 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" event={"ID":"48ec31e2-b484-45a8-9888-35c505e1752b","Type":"ContainerStarted","Data":"4aacb30c4c88a8ba05bcd50e311af5d703d94bfddbeaf422c374f7916311ce57"} Oct 30 21:37:27.703443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:27.703388 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podStartSLOduration=2.688632922 podStartE2EDuration="7.70337226s" totalImagesPullingTime="5.014739338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:27.700508834 +0000 UTC m=+509.896199956" watchObservedRunningTime="2026-10-30 21:37:27.70337226 +0000 UTC m=+509.899063379" Oct 30 21:37:27.982973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:27.976298 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:37:27.989132 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:27.987413 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:37:28.298810 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:28.298733 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:28.584675 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:28.584587 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:37:28.585629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:28.585604 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:37:31.299153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:31.299115 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:31.312292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:31.312268 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:31.718651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:31.718616 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:37:33.354715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.354672 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc"] Oct 30 21:37:33.363516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.363485 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.366799 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.366764 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc"] Oct 30 21:37:33.380787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.379259 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"default-dockercfg-6lxnt\"" Oct 30 21:37:33.380787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.379434 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"simulator-w1-1a81ae-kserve-self-signed-certs\"" Oct 30 21:37:33.380787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.379546 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"openshift-service-ca.crt\"" Oct 30 21:37:33.380787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.379694 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"kube-root-ca.crt\"" Oct 30 21:37:33.435228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.435196 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9h4r\" (UniqueName: \"kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.435415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.435262 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.435415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.435391 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.435521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.435490 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.435652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.435633 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.435722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.435690 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.536994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.536933 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.537194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.537018 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.537194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.537085 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.537194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.537155 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.537194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.537194 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.537407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.537227 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9h4r\" (UniqueName: \"kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.538468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.538441 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.538722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.538693 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.538848 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.538801 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.543621 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.543595 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.549049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.545204 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.551249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.551192 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9h4r\" (UniqueName: \"kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r\") pod \"simulator-w1-1a81ae-kserve-7569557765-g2tfc\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:33.687973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:33.684164 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:34.067174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:34.067097 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc"] Oct 30 21:37:34.070608 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:34.070580 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d9f10f5_6db7_4b17_b551_69f7bed48737.slice/crio-b6644de59d8cb6eaf6721cb9cd931665c9c6c134abcd138cddf958013995ffc9 WatchSource:0}: Error finding container b6644de59d8cb6eaf6721cb9cd931665c9c6c134abcd138cddf958013995ffc9: Status 404 returned error can't find the container with id b6644de59d8cb6eaf6721cb9cd931665c9c6c134abcd138cddf958013995ffc9 Oct 30 21:37:34.759001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:34.758935 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" event={"ID":"8d9f10f5-6db7-4b17-b551-69f7bed48737","Type":"ContainerStarted","Data":"e2c7420d2a30c9f9eabaa3b2f1a631687599c9d90aeb67eacec9758412e05ae4"} Oct 30 21:37:34.759001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:34.759003 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" event={"ID":"8d9f10f5-6db7-4b17-b551-69f7bed48737","Type":"ContainerStarted","Data":"b6644de59d8cb6eaf6721cb9cd931665c9c6c134abcd138cddf958013995ffc9"} Oct 30 21:37:35.684695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:35.684663 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:35.701214 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:37:35.701179 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:37:39.703207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.703153 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podStartSLOduration=6.703137825 podStartE2EDuration="6.703137825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:34.825134062 +0000 UTC m=+517.020825181" watchObservedRunningTime="2026-10-30 21:37:39.703137825 +0000 UTC m=+521.898828940" Oct 30 21:37:39.703595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.703432 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj"] Oct 30 21:37:39.707697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.707671 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.717604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.717580 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"default-dockercfg-ldhl4\"" Oct 30 21:37:39.723794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.723772 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"simulator-w2-db0b54-kserve-self-signed-certs\"" Oct 30 21:37:39.725356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.725325 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"kube-root-ca.crt\"" Oct 30 21:37:39.725356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.725338 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"openshift-service-ca.crt\"" Oct 30 21:37:39.727929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.727903 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj"] Oct 30 21:37:39.820873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.820840 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.820873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.820874 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.821156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.820908 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.821156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.821002 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.821156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.821124 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.821327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.821194 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pksp\" (UniqueName: \"kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.924634 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.924747 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pksp\" (UniqueName: \"kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.924810 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.924840 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.924890 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.924916 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.925746 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.926003 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.932981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.932249 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.934126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.934100 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.934527 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.934505 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:39.947456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:39.947428 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pksp\" (UniqueName: \"kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp\") pod \"simulator-w2-db0b54-kserve-856f455cd6-jvgcj\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:40.021388 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:40.021312 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:40.367238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:40.367118 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj"] Oct 30 21:37:40.370039 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:40.369997 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9540531_1013_451b_b259_6fcf2ad094fb.slice/crio-1431c4163844812bf5ab6e61fd815b8a960a52e8c9349a827e45646409716c6b WatchSource:0}: Error finding container 1431c4163844812bf5ab6e61fd815b8a960a52e8c9349a827e45646409716c6b: Status 404 returned error can't find the container with id 1431c4163844812bf5ab6e61fd815b8a960a52e8c9349a827e45646409716c6b Oct 30 21:37:40.811974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:40.802393 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" event={"ID":"e9540531-1013-451b-b259-6fcf2ad094fb","Type":"ContainerStarted","Data":"d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae"} Oct 30 21:37:40.811974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:40.802458 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" event={"ID":"e9540531-1013-451b-b259-6fcf2ad094fb","Type":"ContainerStarted","Data":"1431c4163844812bf5ab6e61fd815b8a960a52e8c9349a827e45646409716c6b"} Oct 30 21:37:40.836998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:40.832722 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podStartSLOduration=1.832704678 podStartE2EDuration="1.832704678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:40.831261246 +0000 UTC m=+523.026952366" watchObservedRunningTime="2026-10-30 21:37:40.832704678 +0000 UTC m=+523.028395798" Oct 30 21:37:41.022459 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:41.022428 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:43.476825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.476789 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz"] Oct 30 21:37:43.481693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.481671 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.512366 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.512335 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"kube-root-ca.crt\"" Oct 30 21:37:43.512630 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.512613 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"default-dockercfg-jldb5\"" Oct 30 21:37:43.512855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.512840 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"premium-w4-0fa864-kserve-self-signed-certs\"" Oct 30 21:37:43.515793 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.515763 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"openshift-service-ca.crt\"" Oct 30 21:37:43.520606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.520567 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz"] Oct 30 21:37:43.571670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.571639 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.571830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.571696 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.571830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.571726 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.571904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.571896 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.571938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.571922 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.572000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.571963 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg5n7\" (UniqueName: \"kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.675849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.675807 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.676095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.675859 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.676095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.675885 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg5n7\" (UniqueName: \"kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.676095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.675994 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.676095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.676030 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.676095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.676051 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.677097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.677064 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.677541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.677503 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.677800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.677781 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.683866 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.683839 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.683996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.683914 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.684819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.684793 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:43.690254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.690231 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg5n7\" (UniqueName: \"kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7\") pod \"premium-w4-0fa864-kserve-5b7fd4b589-jr4zz\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.699400 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.699377 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:43.793829 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.793756 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:43.846528 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:43.845692 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:37:44.079502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:44.079470 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz"] Oct 30 21:37:44.080084 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:44.080059 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf885f90c_9fa9_455b_ad81_5225b47fb9f9.slice/crio-4a2f08af183ec7f8166e35eec355e8da1361e5748c81b09703a8aeedc0efcbf6 WatchSource:0}: Error finding container 4a2f08af183ec7f8166e35eec355e8da1361e5748c81b09703a8aeedc0efcbf6: Status 404 returned error can't find the container with id 4a2f08af183ec7f8166e35eec355e8da1361e5748c81b09703a8aeedc0efcbf6 Oct 30 21:37:44.831918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:44.831877 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" event={"ID":"f885f90c-9fa9-455b-ad81-5225b47fb9f9","Type":"ContainerStarted","Data":"364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3"} Oct 30 21:37:44.832496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:44.831926 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" event={"ID":"f885f90c-9fa9-455b-ad81-5225b47fb9f9","Type":"ContainerStarted","Data":"4a2f08af183ec7f8166e35eec355e8da1361e5748c81b09703a8aeedc0efcbf6"} Oct 30 21:37:44.870966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:44.870904 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podStartSLOduration=1.8708913809999999 podStartE2EDuration="1.870891381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:44.870172613 +0000 UTC m=+527.065863724" watchObservedRunningTime="2026-10-30 21:37:44.870891381 +0000 UTC m=+527.066582499" Oct 30 21:37:45.794858 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:45.794830 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:50.021673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:50.021638 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:50.040556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:50.040531 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:50.901758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:50.901729 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:37:53.310442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.310397 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k"] Oct 30 21:37:53.314759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.314733 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.323062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.323041 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"premium-w0-000f64-kserve-self-signed-certs\"" Oct 30 21:37:53.325967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.325451 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k"] Oct 30 21:37:53.384475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.384441 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.384666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.384580 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.384666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.384623 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.384785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.384727 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxnj\" (UniqueName: \"kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.384842 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.384804 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.384896 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.384864 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.485975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.485545 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.485975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.485631 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.485975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.485684 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.485975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.485772 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.485975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.485796 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.485975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.485854 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxnj\" (UniqueName: \"kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.488004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.486747 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.488004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.487038 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.488004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.487610 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.510693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.510650 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxnj\" (UniqueName: \"kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.512032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.512002 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.512919 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.512891 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs\") pod \"premium-w0-000f64-kserve-5758cf7c78-6dw8k\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.633171 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.633146 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:37:53.794548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.794519 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:53.806751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.806727 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:53.913672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.913640 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k"] Oct 30 21:37:53.915900 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:37:53.915872 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec09683_4e3f_4351_9faf_9cccccd36f9f.slice/crio-bf5d1336ffc1f27680f22f58fb17aa37e8dc8e54771838cc03ff5591af859e6f WatchSource:0}: Error finding container bf5d1336ffc1f27680f22f58fb17aa37e8dc8e54771838cc03ff5591af859e6f: Status 404 returned error can't find the container with id bf5d1336ffc1f27680f22f58fb17aa37e8dc8e54771838cc03ff5591af859e6f Oct 30 21:37:53.927696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:53.927677 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:37:54.927737 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:54.927702 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" event={"ID":"7ec09683-4e3f-4351-9faf-9cccccd36f9f","Type":"ContainerStarted","Data":"145434cc8bc7436f2f98e6f8c210e173a8d638e7be843e61924fe450a13c5c53"} Oct 30 21:37:54.928311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:54.928288 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" event={"ID":"7ec09683-4e3f-4351-9faf-9cccccd36f9f","Type":"ContainerStarted","Data":"bf5d1336ffc1f27680f22f58fb17aa37e8dc8e54771838cc03ff5591af859e6f"} Oct 30 21:37:54.961969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:54.959031 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podStartSLOduration=1.9590025450000002 podStartE2EDuration="1.959002545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:54.956303766 +0000 UTC m=+537.151994886" watchObservedRunningTime="2026-10-30 21:37:54.959002545 +0000 UTC m=+537.154693665" Oct 30 21:37:55.633375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:37:55.633332 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:38:03.633878 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:03.633831 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:38:03.647643 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:03.647615 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:38:04.037711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:04.037629 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:38:05.698369 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:38:05.698330 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:38:07.983247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:07.983202 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w"] Oct 30 21:38:07.989191 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:07.989162 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:07.994136 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:07.994115 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0fa864-openshift-default-dockercfg-bwzs2\"" Oct 30 21:38:07.999713 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:07.999687 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w"] Oct 30 21:38:08.089884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.089849 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.089894 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.089919 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.090057 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.090092 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.090121 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.090202 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcj9b\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.090251 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.090459 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.090270 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.191980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.191922 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192599 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192368 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192592 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192809 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192855 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192896 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192895 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192924 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.192996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.192987 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcj9b\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.193279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.193053 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.193279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.193079 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.193279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.193206 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.193997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.193973 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.194119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.194040 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.196107 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.196087 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.196545 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.196512 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.213152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.209089 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.213152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.211750 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcj9b\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.303917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.303844 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:08.535009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.534066 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg"] Oct 30 21:38:08.540048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.540006 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.555049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.552800 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1a81ae-openshift-default-dockercfg-mbfh7\"" Oct 30 21:38:08.560818 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.560730 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg"] Oct 30 21:38:08.608309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608274 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608322 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608342 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608486 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608553 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608649 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608607 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608695 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608786 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.608864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.608819 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmt27\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.672196 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.671879 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w"] Oct 30 21:38:08.673963 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:38:08.673920 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2439ab7_e15d_4bbf_b6b1_affdb57e26f8.slice/crio-1e351c9674b4ad461c9bbdd1bb9fcdea17dc5a50bf6e9d41f423de7804e92634 WatchSource:0}: Error finding container 1e351c9674b4ad461c9bbdd1bb9fcdea17dc5a50bf6e9d41f423de7804e92634: Status 404 returned error can't find the container with id 1e351c9674b4ad461c9bbdd1bb9fcdea17dc5a50bf6e9d41f423de7804e92634 Oct 30 21:38:08.676478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.676444 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:08.676563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.676512 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:08.676563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.676541 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:08.709630 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709598 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.709738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709665 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.709738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709707 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709739 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709803 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709825 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmt27\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709887 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709935 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.709979 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.710036 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.710326 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.710409 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.710484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.710484 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.711032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.710503 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.712855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.712828 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.712971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.712893 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.722773 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.722752 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.722863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.722831 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmt27\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:08.857805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:08.857720 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:09.030041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.029993 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" event={"ID":"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8","Type":"ContainerStarted","Data":"67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0"} Oct 30 21:38:09.030457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.030060 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" event={"ID":"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8","Type":"ContainerStarted","Data":"1e351c9674b4ad461c9bbdd1bb9fcdea17dc5a50bf6e9d41f423de7804e92634"} Oct 30 21:38:09.056715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.056660 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podStartSLOduration=2.056642775 podStartE2EDuration="2.056642775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:09.053387817 +0000 UTC m=+551.249078964" watchObservedRunningTime="2026-10-30 21:38:09.056642775 +0000 UTC m=+551.252333893" Oct 30 21:38:09.145349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.145320 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg"] Oct 30 21:38:09.147923 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:38:09.147897 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6e6998_bee0_47aa_9fe6_87168a547b1e.slice/crio-cd03ae3076d8c1fda167fca2fd276455a9ca6201c3145104f04998174c156f7e WatchSource:0}: Error finding container cd03ae3076d8c1fda167fca2fd276455a9ca6201c3145104f04998174c156f7e: Status 404 returned error can't find the container with id cd03ae3076d8c1fda167fca2fd276455a9ca6201c3145104f04998174c156f7e Oct 30 21:38:09.150497 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.150457 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:09.150595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.150536 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:09.150595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.150566 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:09.305012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.304976 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:09.306094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.306064 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:09.306247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:09.306118 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:10.066971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.063908 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" event={"ID":"7d6e6998-bee0-47aa-9fe6-87168a547b1e","Type":"ContainerStarted","Data":"66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926"} Oct 30 21:38:10.066971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.064005 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" event={"ID":"7d6e6998-bee0-47aa-9fe6-87168a547b1e","Type":"ContainerStarted","Data":"cd03ae3076d8c1fda167fca2fd276455a9ca6201c3145104f04998174c156f7e"} Oct 30 21:38:10.094020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.093568 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podStartSLOduration=2.093547734 podStartE2EDuration="2.093547734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:10.08833186 +0000 UTC m=+552.284022981" watchObservedRunningTime="2026-10-30 21:38:10.093547734 +0000 UTC m=+552.289238858" Oct 30 21:38:10.304974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.304516 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:10.304974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.304631 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:10.871974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.869860 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:10.871974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.870011 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:38:10.871974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:10.870056 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:38:11.307131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:11.307090 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:11.307663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:11.307169 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:11.858340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:11.858301 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:38:11.858633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:11.858603 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:38:12.305407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:12.305365 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:12.305737 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:12.305707 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:12.859423 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:12.859385 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:38:12.860049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:12.859999 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:38:13.304806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:13.304756 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:13.305040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:13.304828 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:13.862123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:13.861354 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:38:13.862123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:13.861413 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:38:14.307973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:14.307309 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:14.307973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:14.307381 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:14.858429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:14.858389 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:38:14.858726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:14.858697 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:38:15.304473 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:15.304427 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:38:15.305127 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:15.304499 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:38:15.858806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:15.858764 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:38:15.858996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:15.858834 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:38:16.710644 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:16.710523 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:16.712041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:16.712017 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:16.714063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:16.714045 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:38:17.061007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:17.060878 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:17.061310 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:17.061290 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:17.062453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:17.062427 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:38:23.854361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:23.854321 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d"] Oct 30 21:38:23.859198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:23.859168 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:23.863575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:23.863549 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-000f64-openshift-default-dockercfg-8l2xx\"" Oct 30 21:38:23.872360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:23.872332 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d"] Oct 30 21:38:24.010735 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.010630 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.018215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.018172 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tdv\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.018482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.018457 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.018625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.018595 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.018860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.018836 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.019112 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.019079 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.019298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.019280 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.019431 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.019413 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.019609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.019591 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121155 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121231 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121278 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tdv\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121294 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121308 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121354 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121371 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121398 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121416 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121649 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121887 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.121992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.121969 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.122039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.122025 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.122252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.122228 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.123778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.123749 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.123942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.123924 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.133868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.133845 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tdv\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.134058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.134027 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.179774 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.179741 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:24.464011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.463982 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d"] Oct 30 21:38:24.467043 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:38:24.467006 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc2476c_7f34_4ed3_b880_b797ee00aa02.slice/crio-47238b326a963c88cb65b90ce8ba76209393e1b0c945a992fef97e6d514545e8 WatchSource:0}: Error finding container 47238b326a963c88cb65b90ce8ba76209393e1b0c945a992fef97e6d514545e8: Status 404 returned error can't find the container with id 47238b326a963c88cb65b90ce8ba76209393e1b0c945a992fef97e6d514545e8 Oct 30 21:38:24.469804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.469765 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:24.469890 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.469848 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:24.469940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:24.469891 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:25.269847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:25.268793 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" event={"ID":"3cc2476c-7f34-4ed3-b880-b797ee00aa02","Type":"ContainerStarted","Data":"c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3"} Oct 30 21:38:25.269847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:25.268864 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" event={"ID":"3cc2476c-7f34-4ed3-b880-b797ee00aa02","Type":"ContainerStarted","Data":"47238b326a963c88cb65b90ce8ba76209393e1b0c945a992fef97e6d514545e8"} Oct 30 21:38:25.295624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:25.295570 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podStartSLOduration=2.295556038 podStartE2EDuration="2.295556038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:25.293370352 +0000 UTC m=+567.489061483" watchObservedRunningTime="2026-10-30 21:38:25.295556038 +0000 UTC m=+567.491247241" Oct 30 21:38:26.180463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:26.180427 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:26.182257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:26.182229 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:38:26.182360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:26.182287 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:38:27.181025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:27.180984 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:38:27.181497 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:27.181058 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:38:28.180125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:28.180083 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:38:28.180298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:28.180144 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:38:29.180920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:29.180881 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:38:29.181355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:29.180978 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:38:30.180504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:30.180451 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:38:30.180682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:30.180542 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:38:31.409521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:31.409424 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:31.410065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:31.410027 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:31.411219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:38:31.411195 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:38:35.714602 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:38:35.714533 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:39:05.699130 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:39:05.699086 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:39:08.440910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.440868 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm"] Oct 30 21:39:08.446472 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.446440 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.454881 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.454855 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7dfff0-openshift-default-dockercfg-lrx4l\"" Oct 30 21:39:08.459126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.459105 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm"] Oct 30 21:39:08.528563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528528 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hzv\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.528750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528577 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.528750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528658 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.528750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528700 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.528876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528800 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.528912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528872 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.528962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528918 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.529003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.528958 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.529037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.529021 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630151 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630122 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630159 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630188 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630225 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630381 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hzv\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630449 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630489 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630521 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630588 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630626 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630642 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.630893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.630868 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.631054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.631031 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.631475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.631449 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.632661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.632633 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.633333 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.633312 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.650788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.650761 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hzv\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.650882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.650789 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:08.764043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:08.763963 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:09.048989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.048966 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm"] Oct 30 21:39:09.051025 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:39:09.050996 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415506c4_eeaf_407f_8562_bd837602183d.slice/crio-969336d1338d7ab77b3a10f0dadb940fd478f7c0b98c1128df6630351a16c763 WatchSource:0}: Error finding container 969336d1338d7ab77b3a10f0dadb940fd478f7c0b98c1128df6630351a16c763: Status 404 returned error can't find the container with id 969336d1338d7ab77b3a10f0dadb940fd478f7c0b98c1128df6630351a16c763 Oct 30 21:39:09.053340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.053305 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:09.053440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.053369 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:09.053440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.053399 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:09.597055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.597014 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" event={"ID":"415506c4-eeaf-407f-8562-bd837602183d","Type":"ContainerStarted","Data":"5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6"} Oct 30 21:39:09.597549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.597087 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" event={"ID":"415506c4-eeaf-407f-8562-bd837602183d","Type":"ContainerStarted","Data":"969336d1338d7ab77b3a10f0dadb940fd478f7c0b98c1128df6630351a16c763"} Oct 30 21:39:09.622624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.622575 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podStartSLOduration=1.622560121 podStartE2EDuration="1.622560121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:09.620518233 +0000 UTC m=+611.816209351" watchObservedRunningTime="2026-10-30 21:39:09.622560121 +0000 UTC m=+611.818251240" Oct 30 21:39:09.771973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:09.769043 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:10.774513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:10.774263 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:39:10.774513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:10.774355 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:39:11.768974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:11.768278 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:39:11.768974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:11.768354 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:39:12.769168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:12.767322 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:39:12.769168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:12.767402 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:39:13.768977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:13.768366 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:39:13.768977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:13.768442 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:39:14.766481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:14.766443 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:39:14.767026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:14.766509 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:39:16.766016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:16.765933 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:16.766511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:16.766043 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.811090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:17.811049 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:17.811700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:17.811665 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:18.692317 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:18.692283 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:18.693999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:18.693978 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:18.709847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:18.708846 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:39:22.164128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.164093 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k"] Oct 30 21:39:22.168211 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.168190 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.182195 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.182173 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:22.183582 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.183561 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:39:22.192984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.192942 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k"] Oct 30 21:39:22.319338 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.319299 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.319338 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.319336 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.319595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.319418 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.420010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.419917 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.420010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.419972 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.420225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.420038 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.420764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.420744 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.422498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.422471 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.433709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.433680 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj\") pod \"maas-api-revoke-sub-ebc981ee-gcm5k\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.482914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.482890 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:22.763434 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:22.763261 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k"] Oct 30 21:39:22.765779 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:39:22.765742 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e8d17e_eadc_49d8_8219_4ebbc729e78e.slice/crio-5726c8ee6f6999d98990eab33b763d0876cf3e21813060da3b3119a2e432b4e6 WatchSource:0}: Error finding container 5726c8ee6f6999d98990eab33b763d0876cf3e21813060da3b3119a2e432b4e6: Status 404 returned error can't find the container with id 5726c8ee6f6999d98990eab33b763d0876cf3e21813060da3b3119a2e432b4e6 Oct 30 21:39:23.744736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:23.744698 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" event={"ID":"e1e8d17e-eadc-49d8-8219-4ebbc729e78e","Type":"ContainerStarted","Data":"5726c8ee6f6999d98990eab33b763d0876cf3e21813060da3b3119a2e432b4e6"} Oct 30 21:39:26.768254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:26.768218 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" event={"ID":"e1e8d17e-eadc-49d8-8219-4ebbc729e78e","Type":"ContainerStarted","Data":"4d4f33e924ad4ec4ba4753175268c36b20b0d5ddc616c4840be705761e659d3d"} Oct 30 21:39:26.789669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:26.789608 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" podStartSLOduration=1.5505886960000002 podStartE2EDuration="4.78959089s" totalImagesPullingTime="3.239002194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:26.785571108 +0000 UTC m=+628.981262241" watchObservedRunningTime="2026-10-30 21:39:26.78959089 +0000 UTC m=+628.985282010" Oct 30 21:39:27.781135 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:27.781098 6808 generic.go:356] "Generic (PLEG): container finished" podID="e1e8d17e-eadc-49d8-8219-4ebbc729e78e" containerID="4d4f33e924ad4ec4ba4753175268c36b20b0d5ddc616c4840be705761e659d3d" exitCode=0 Oct 30 21:39:27.781739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:27.781210 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" event={"ID":"e1e8d17e-eadc-49d8-8219-4ebbc729e78e","Type":"ContainerDied","Data":"4d4f33e924ad4ec4ba4753175268c36b20b0d5ddc616c4840be705761e659d3d"} Oct 30 21:39:28.952457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:28.952427 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:29.018253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.018218 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj\") pod \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " Oct 30 21:39:29.018447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.018413 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca\") pod \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " Oct 30 21:39:29.018523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.018458 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca\") pod \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\" (UID: \"e1e8d17e-eadc-49d8-8219-4ebbc729e78e\") " Oct 30 21:39:29.018989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.018801 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca" pod "e1e8d17e-eadc-49d8-8219-4ebbc729e78e" (UID: "e1e8d17e-eadc-49d8-8219-4ebbc729e78e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:29.018989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.018942 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:29.020488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.020458 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj" pod "e1e8d17e-eadc-49d8-8219-4ebbc729e78e" (UID: "e1e8d17e-eadc-49d8-8219-4ebbc729e78e"). InnerVolumeSpecName "kube-api-access-wlzbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:29.021150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.021129 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca" pod "e1e8d17e-eadc-49d8-8219-4ebbc729e78e" (UID: "e1e8d17e-eadc-49d8-8219-4ebbc729e78e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:29.122023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.121865 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzbj\" (UniqueName: \"kubernetes.io/projected/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-kube-api-access-wlzbj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:29.122023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.121911 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e8d17e-eadc-49d8-8219-4ebbc729e78e-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:29.829974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.815125 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" event={"ID":"e1e8d17e-eadc-49d8-8219-4ebbc729e78e","Type":"ContainerDied","Data":"5726c8ee6f6999d98990eab33b763d0876cf3e21813060da3b3119a2e432b4e6"} Oct 30 21:39:29.829974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.815196 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5726c8ee6f6999d98990eab33b763d0876cf3e21813060da3b3119a2e432b4e6" Oct 30 21:39:29.829974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:29.815272 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k" Oct 30 21:39:35.698118 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:39:35.698076 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:39:46.434397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.434309 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24"] Oct 30 21:39:46.435016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.434842 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e1e8d17e-eadc-49d8-8219-4ebbc729e78e" containerName="revoke-keys" Oct 30 21:39:46.435016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.434862 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e8d17e-eadc-49d8-8219-4ebbc729e78e" containerName="revoke-keys" Oct 30 21:39:46.435170 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.435014 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e8d17e-eadc-49d8-8219-4ebbc729e78e" containerName="revoke-keys" Oct 30 21:39:46.438599 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.438576 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.453571 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.453551 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"distinct-w2-db0b54-kserve-self-signed-certs\"" Oct 30 21:39:46.454720 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.454696 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24"] Oct 30 21:39:46.498489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.498457 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.498657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.498524 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.498707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.498679 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmbz\" (UniqueName: \"kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.498763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.498748 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.498877 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.498858 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.498944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.498898 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600205 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600317 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmbz\" (UniqueName: \"kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600377 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600437 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600806 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600836 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.600994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.600876 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.601119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.601024 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.601389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.601362 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.603738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.603711 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.603928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.603911 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.638989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.629725 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmbz\" (UniqueName: \"kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz\") pod \"distinct-w2-db0b54-kserve-89c8db878-d7r24\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:46.752350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:46.752004 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:47.048269 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:47.048181 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24"] Oct 30 21:39:47.129017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:47.128986 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" event={"ID":"39bc077a-5dc9-419d-8f30-94cb2ea313da","Type":"ContainerStarted","Data":"79619c8150b52af99c492e477672cc2c02215f37cd8614b3588e3941a91f29fa"} Oct 30 21:39:48.136633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:48.136599 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" event={"ID":"39bc077a-5dc9-419d-8f30-94cb2ea313da","Type":"ContainerStarted","Data":"8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0"} Oct 30 21:39:48.163270 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:48.163211 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podStartSLOduration=2.163199314 podStartE2EDuration="2.163199314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:48.163085694 +0000 UTC m=+650.358776815" watchObservedRunningTime="2026-10-30 21:39:48.163199314 +0000 UTC m=+650.358890433" Oct 30 21:39:48.753095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:48.753065 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:50.192664 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.192625 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q"] Oct 30 21:39:50.197443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.197416 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.206044 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.206020 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q"] Oct 30 21:39:50.206486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.206463 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"distinct-w0-000f64-kserve-self-signed-certs\"" Oct 30 21:39:50.250520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.250493 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.250753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.250731 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.250888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.250864 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.251060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.250903 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.251169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.251149 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.251257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.251235 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzfh\" (UniqueName: \"kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.355657 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.355743 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzfh\" (UniqueName: \"kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.355831 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.355864 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.355898 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.355931 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.357112 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.357749 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.358978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.358052 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.420974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.420077 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.432970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.429087 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.432970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.430852 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzfh\" (UniqueName: \"kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh\") pod \"distinct-w0-000f64-kserve-6744b7b5df-ld78q\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.519146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.519069 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:50.813839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:50.813816 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q"] Oct 30 21:39:50.815676 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:39:50.815642 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod493dcec4_f99e_4ba5_9db1_87868c28af56.slice/crio-eb5613498d15eb2508b7576d310f7d941de9828c8411b3981d990212442a9645 WatchSource:0}: Error finding container eb5613498d15eb2508b7576d310f7d941de9828c8411b3981d990212442a9645: Status 404 returned error can't find the container with id eb5613498d15eb2508b7576d310f7d941de9828c8411b3981d990212442a9645 Oct 30 21:39:51.159482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:51.159446 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" event={"ID":"493dcec4-f99e-4ba5-9db1-87868c28af56","Type":"ContainerStarted","Data":"d601ceed50595c9e926212421e2bf814baac0dab4cf68da822879a2a23e00b64"} Oct 30 21:39:51.159680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:51.159496 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" event={"ID":"493dcec4-f99e-4ba5-9db1-87868c28af56","Type":"ContainerStarted","Data":"eb5613498d15eb2508b7576d310f7d941de9828c8411b3981d990212442a9645"} Oct 30 21:39:51.183457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:51.183394 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podStartSLOduration=1.18337626 podStartE2EDuration="1.18337626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:51.178729986 +0000 UTC m=+653.374421103" watchObservedRunningTime="2026-10-30 21:39:51.18337626 +0000 UTC m=+653.379067381" Oct 30 21:39:51.519373 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:51.519286 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:39:53.606889 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.606848 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk"] Oct 30 21:39:53.611572 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.611542 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.633000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.632968 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk"] Oct 30 21:39:53.701986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.701941 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v"] Oct 30 21:39:53.706088 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.706057 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.721503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.721437 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ngt\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.734790 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.734742 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.734959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.734798 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.734959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.734855 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.734959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.734880 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.734959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.734902 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.735214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.735020 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.735214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.735071 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.735214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.735209 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.738111 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.738042 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v"] Oct 30 21:39:53.837100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837072 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837114 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837138 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837194 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wqf\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837225 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837517 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837378 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837517 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837407 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837517 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837449 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837550 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ngt\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837576 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837578 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837606 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837642 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837716 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.837914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837772 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.837914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837799 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.838098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.837987 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.838098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.838024 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.838098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.838057 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.838253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.838191 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.838305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.838277 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.838441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.838357 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.838589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.838565 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.840537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.840510 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.841248 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.841222 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.853499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.853471 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.857103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.857018 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ngt\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.935557 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.935525 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:53.939110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939087 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939145 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939184 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wqf\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939207 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939226 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939259 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939303 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939341 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939365 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939484 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.939645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939574 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.940229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939674 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.940229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.939869 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.940229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.940132 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.941816 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.941787 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.942245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.942221 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.952815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.952717 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:53.952815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:53.952769 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wqf\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:54.042190 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.038224 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:54.272590 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:39:54.272555 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5982a75_71df_4044_bc2c_a331e4a6cd28.slice/crio-9857794a25b13adec564dc823a3551a5ae8713761418d77a108236e99903701c WatchSource:0}: Error finding container 9857794a25b13adec564dc823a3551a5ae8713761418d77a108236e99903701c: Status 404 returned error can't find the container with id 9857794a25b13adec564dc823a3551a5ae8713761418d77a108236e99903701c Oct 30 21:39:54.272734 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.272659 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk"] Oct 30 21:39:54.275447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.275401 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:54.275561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.275485 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:54.275561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.275530 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:54.373963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.373920 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v"] Oct 30 21:39:54.378217 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:39:54.378161 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb0c9aa_72a0_40ab_a90f_869389caa158.slice/crio-32c35c91ad001e9eb3448e019b02977a28d26d876d891d1c81a38dab4b1421ae WatchSource:0}: Error finding container 32c35c91ad001e9eb3448e019b02977a28d26d876d891d1c81a38dab4b1421ae: Status 404 returned error can't find the container with id 32c35c91ad001e9eb3448e019b02977a28d26d876d891d1c81a38dab4b1421ae Oct 30 21:39:54.380888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.380850 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:54.384100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.380914 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:54.384100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:54.380940 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:39:55.304319 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.304281 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" event={"ID":"b5982a75-71df-4044-bc2c-a331e4a6cd28","Type":"ContainerStarted","Data":"77f6c9bd1297048dedcb39edf29b557fbd9f491cd68bf48a321ddc8aa57eec34"} Oct 30 21:39:55.304937 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.304897 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" event={"ID":"b5982a75-71df-4044-bc2c-a331e4a6cd28","Type":"ContainerStarted","Data":"9857794a25b13adec564dc823a3551a5ae8713761418d77a108236e99903701c"} Oct 30 21:39:55.308638 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.308610 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" event={"ID":"edb0c9aa-72a0-40ab-a90f-869389caa158","Type":"ContainerStarted","Data":"adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3"} Oct 30 21:39:55.308805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.308790 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" event={"ID":"edb0c9aa-72a0-40ab-a90f-869389caa158","Type":"ContainerStarted","Data":"32c35c91ad001e9eb3448e019b02977a28d26d876d891d1c81a38dab4b1421ae"} Oct 30 21:39:55.345744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.345679 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podStartSLOduration=2.345640594 podStartE2EDuration="2.345640594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:55.344935841 +0000 UTC m=+657.540626961" watchObservedRunningTime="2026-10-30 21:39:55.345640594 +0000 UTC m=+657.541331698" Oct 30 21:39:55.377187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.377128 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podStartSLOduration=2.377110003 podStartE2EDuration="2.377110003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:55.372067132 +0000 UTC m=+657.567758249" watchObservedRunningTime="2026-10-30 21:39:55.377110003 +0000 UTC m=+657.572801124" Oct 30 21:39:55.948423 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.948085 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:55.948423 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.948145 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:39:55.966041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:55.965943 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:39:56.044797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:56.044725 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:39:56.063975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:56.050086 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.064563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:56.064268 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:39:56.770973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:56.759027 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:56.946396 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:56.946353 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.946601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:56.946434 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:39:57.004078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.004033 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.049031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.048384 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.049031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.048450 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:39:57.736970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.736620 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.748972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.737649 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:57.833094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.833042 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.937973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.893069 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.980569 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.977514 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.980569 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.977570 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:39:57.986686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:57.985483 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.006102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.006016 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:39:58.054566 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.054533 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.054821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.054794 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:39:58.140183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.140152 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:39:58.738987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.738076 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.738987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.738525 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.738987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.738749 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:39:58.830785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.830753 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:39:58.864975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.854462 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:39:58.867067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.867022 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.868037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.868013 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:39:58.937602 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.937564 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.937781 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.937625 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:39:58.939565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.939544 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:39:58.995192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.995096 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.000969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:58.996017 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:39:59.013091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.013052 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.047177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.047143 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.047430 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.047405 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:39:59.127309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.126683 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:39:59.133064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.130982 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:39:59.935965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.935908 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.936395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:39:59.936370 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:40:00.039016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.038979 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:00.039184 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.039038 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:40:00.523394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.523360 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:40:00.626316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.626292 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:40:00.735470 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.735439 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:40:00.936208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.936172 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:40:00.936711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:00.936220 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:40:01.039389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:01.039083 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.039389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:01.039138 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:40:01.936756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:01.936722 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.937342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:01.937310 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:40:02.041974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:02.039672 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.041974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:02.039730 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:40:03.039595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:03.039550 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:03.040437 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:03.040312 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:40:03.947989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:03.944449 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:03.947989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:03.944511 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:04.782442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:04.778028 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:40:04.782442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:04.779322 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:40:04.783292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:04.783271 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:40:04.981789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:04.981736 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:04.982016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:04.981814 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:05.034794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.034714 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:40:05.035987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.035962 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:40:05.036856 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.036832 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:40:05.433535 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.433488 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4"] Oct 30 21:40:05.442060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.442036 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.448170 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.448140 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"distinct-w1-1a81ae-kserve-self-signed-certs\"" Oct 30 21:40:05.448732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.448704 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4"] Oct 30 21:40:05.597972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.593817 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.597972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.594000 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.597972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.594057 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.597972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.594124 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.597972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.594152 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w775\" (UniqueName: \"kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.597972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.594283 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.699080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.698982 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.699336 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.699317 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.699498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.699483 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.699614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.699597 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w775\" (UniqueName: \"kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.699810 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.699785 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.699887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.699796 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.700028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.700013 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.700460 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.700439 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.701220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.701196 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.723197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.722395 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.730196 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:40:05.730162 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:40:05.733428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.731255 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.733428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.731850 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w775\" (UniqueName: \"kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775\") pod \"distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:05.759540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:05.759351 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:06.057447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:06.057421 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4"] Oct 30 21:40:06.058802 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:06.058774 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c4d339a_ae3c_42c1_85fe_cd80be3e2a97.slice/crio-ed14a136efd51d9dfde13eee6801dc95279ee5838cb24f2caae422e79e6adab8 WatchSource:0}: Error finding container ed14a136efd51d9dfde13eee6801dc95279ee5838cb24f2caae422e79e6adab8: Status 404 returned error can't find the container with id ed14a136efd51d9dfde13eee6801dc95279ee5838cb24f2caae422e79e6adab8 Oct 30 21:40:06.774466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:06.774432 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" event={"ID":"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97","Type":"ContainerStarted","Data":"37498662991cfee98111914bc3555167d5f0668465dfa9141d687cede4ecc7ac"} Oct 30 21:40:06.774644 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:06.774478 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" event={"ID":"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97","Type":"ContainerStarted","Data":"ed14a136efd51d9dfde13eee6801dc95279ee5838cb24f2caae422e79e6adab8"} Oct 30 21:40:06.796283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:06.796236 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podStartSLOduration=1.7962197789999998 podStartE2EDuration="1.796219779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:06.794020372 +0000 UTC m=+668.989711493" watchObservedRunningTime="2026-10-30 21:40:06.796219779 +0000 UTC m=+668.991910899" Oct 30 21:40:07.761286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:07.761256 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:08.314458 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.314418 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6"] Oct 30 21:40:08.349499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.349465 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6"] Oct 30 21:40:08.349739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.349647 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.374925 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.374897 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-caa3dc-openshift-default-dockercfg-5kw62\"" Oct 30 21:40:08.449479 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.449448 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.449668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.449493 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.449668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.449518 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.449808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.449782 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.449874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.449826 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.450006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.449990 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.450080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.450025 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.450080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.450062 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.450169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.450143 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zww\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.550679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.550643 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.550846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.550746 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.550846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.550773 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.550846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.550842 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.550872 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.550925 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551009 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zww\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551058 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551101 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551315 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551286 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551362 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551517 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551650 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551618 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.551935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.551914 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.553595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.553574 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.553738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.553721 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.564218 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.564190 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.564315 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.564298 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zww\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.690663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.690622 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:08.987428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.987284 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6"] Oct 30 21:40:08.989971 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:08.989919 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f56aa7_dc0b_46df_89eb_eff7d729c6b1.slice/crio-1aac186a74d7d81bea77e3b52694ed454fda7695dd69af4c3cdf7158ac818dfe WatchSource:0}: Error finding container 1aac186a74d7d81bea77e3b52694ed454fda7695dd69af4c3cdf7158ac818dfe: Status 404 returned error can't find the container with id 1aac186a74d7d81bea77e3b52694ed454fda7695dd69af4c3cdf7158ac818dfe Oct 30 21:40:08.993046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.992989 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:08.993143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.993089 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:08.993143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:08.993122 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:09.800886 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:09.800843 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" event={"ID":"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1","Type":"ContainerStarted","Data":"44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316"} Oct 30 21:40:09.801125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:09.800916 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" event={"ID":"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1","Type":"ContainerStarted","Data":"1aac186a74d7d81bea77e3b52694ed454fda7695dd69af4c3cdf7158ac818dfe"} Oct 30 21:40:09.834764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:09.834699 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podStartSLOduration=1.8346800189999999 podStartE2EDuration="1.834680019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.833386588 +0000 UTC m=+672.029077719" watchObservedRunningTime="2026-10-30 21:40:09.834680019 +0000 UTC m=+672.030371141" Oct 30 21:40:10.691662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:10.691618 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:10.693801 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:10.693777 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:40:10.693912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:10.693829 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:40:11.691465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:11.691423 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:40:11.691632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:11.691492 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:40:12.691360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:12.691322 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:40:12.691784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:12.691378 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:40:13.691411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:13.691369 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:40:13.691821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:13.691433 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:40:14.690924 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:14.690890 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:40:14.691148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:14.690942 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:40:15.056489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.056391 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p"] Oct 30 21:40:15.068984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.063253 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.075057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.075024 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p"] Oct 30 21:40:15.075533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.075511 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"default-dockercfg-c2fpr\"" Oct 30 21:40:15.076302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.075761 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"openshift-service-ca.crt\"" Oct 30 21:40:15.076435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.075840 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"kube-root-ca.crt\"" Oct 30 21:40:15.079207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.079181 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"distinct-2-w5-caa3dc-kserve-self-signed-certs\"" Oct 30 21:40:15.180708 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.180668 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.181025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.180990 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.181173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.181150 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxmz\" (UniqueName: \"kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.181258 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.181232 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.181360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.181339 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.181471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.181405 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.290974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.290728 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.290974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.290787 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.290974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.290878 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxmz\" (UniqueName: \"kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.290974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.290942 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.291380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.291060 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.291380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.291119 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.298429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.292387 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.298429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.293259 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.298429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.293506 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.301968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.299346 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.314029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.306666 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxmz\" (UniqueName: \"kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.332991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.330591 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs\") pod \"distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.413137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.412964 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:15.691777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.691735 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:40:15.691990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.691797 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:40:15.710772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.710748 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p"] Oct 30 21:40:15.713415 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:15.713387 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ffe4f8_d41c_4236_a34e_19a756febd79.slice/crio-0abe9f7ceac11047245b2c5c23c3e313c7fd8ed9e3e113202e31ee98c3bd1410 WatchSource:0}: Error finding container 0abe9f7ceac11047245b2c5c23c3e313c7fd8ed9e3e113202e31ee98c3bd1410: Status 404 returned error can't find the container with id 0abe9f7ceac11047245b2c5c23c3e313c7fd8ed9e3e113202e31ee98c3bd1410 Oct 30 21:40:15.760886 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.760862 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:15.781532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.781507 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:15.844525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.844493 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" event={"ID":"f2ffe4f8-d41c-4236-a34e-19a756febd79","Type":"ContainerStarted","Data":"f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b"} Oct 30 21:40:15.844703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.844534 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" event={"ID":"f2ffe4f8-d41c-4236-a34e-19a756febd79","Type":"ContainerStarted","Data":"0abe9f7ceac11047245b2c5c23c3e313c7fd8ed9e3e113202e31ee98c3bd1410"} Oct 30 21:40:15.857579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.857552 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:15.867528 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:15.867474 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podStartSLOduration=0.867454696 podStartE2EDuration="867.454696ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:15.865239609 +0000 UTC m=+678.060930728" watchObservedRunningTime="2026-10-30 21:40:15.867454696 +0000 UTC m=+678.063145826" Oct 30 21:40:16.413498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:16.413464 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:16.696063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:16.695994 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:16.696529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:16.696501 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:16.697351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:16.697325 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:40:23.336367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.336336 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c"] Oct 30 21:40:23.340636 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.340607 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.355247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.355221 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c"] Oct 30 21:40:23.472368 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472328 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472372 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472406 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472560 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472740 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472715 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472776 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fqb\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472851 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.472996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.472980 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.473114 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.473022 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.582763 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.582816 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.582852 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.582921 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.583048 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.583096 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fqb\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.583153 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.583253 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.583374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.583296 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.584340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.584308 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.584858 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.584829 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.585130 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.585106 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.585362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.585338 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.585631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.585605 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.592464 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.592392 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.592563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.592522 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.594078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.594059 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b"] Oct 30 21:40:23.598398 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.598379 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.599862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.599840 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.600234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.600198 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fqb\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.617897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.617044 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b"] Oct 30 21:40:23.662994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.661028 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:23.684799 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.684756 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.684923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.684819 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.684923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.684855 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcm7\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.684923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.684901 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.685183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.684933 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.685183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.684992 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.685183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.685089 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.685183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.685128 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.685183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.685151 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.792698 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.792767 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.792792 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.792876 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.792924 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.792980 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcm7\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.793041 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.793077 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.793125 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.793701 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.794368 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.795002 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.826846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.800004 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.832983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.832189 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.832983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.832687 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.836096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.836047 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.870139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.859439 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.893345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.893285 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcm7\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:23.956456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:23.956210 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:24.015760 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.015454 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c"] Oct 30 21:40:24.019431 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:24.019387 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e78663_fd4e_451e_a3f4_380804008a8a.slice/crio-f04e5cc2c724f262251d57a97b8cb5b125e9fe2e8fdd05277e711423f6b99891 WatchSource:0}: Error finding container f04e5cc2c724f262251d57a97b8cb5b125e9fe2e8fdd05277e711423f6b99891: Status 404 returned error can't find the container with id f04e5cc2c724f262251d57a97b8cb5b125e9fe2e8fdd05277e711423f6b99891 Oct 30 21:40:24.024428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.023084 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:24.024428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.023172 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:24.024428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.023215 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:24.035023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.034993 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" event={"ID":"a6e78663-fd4e-451e-a3f4-380804008a8a","Type":"ContainerStarted","Data":"f04e5cc2c724f262251d57a97b8cb5b125e9fe2e8fdd05277e711423f6b99891"} Oct 30 21:40:24.295873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.295808 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b"] Oct 30 21:40:24.298478 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:24.298446 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408895cb_af66_4aa0_9c06_9074d2657bae.slice/crio-40a29c6f25304e2d769e8b80526c66550f05d8a953f6741f9a68de8401fa23f0 WatchSource:0}: Error finding container 40a29c6f25304e2d769e8b80526c66550f05d8a953f6741f9a68de8401fa23f0: Status 404 returned error can't find the container with id 40a29c6f25304e2d769e8b80526c66550f05d8a953f6741f9a68de8401fa23f0 Oct 30 21:40:24.301218 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.301186 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:24.301318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.301246 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:24.301318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:24.301276 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:25.076085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.076044 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" event={"ID":"408895cb-af66-4aa0-9c06-9074d2657bae","Type":"ContainerStarted","Data":"6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd"} Oct 30 21:40:25.076708 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.076114 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" event={"ID":"408895cb-af66-4aa0-9c06-9074d2657bae","Type":"ContainerStarted","Data":"40a29c6f25304e2d769e8b80526c66550f05d8a953f6741f9a68de8401fa23f0"} Oct 30 21:40:25.078042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.078013 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" event={"ID":"a6e78663-fd4e-451e-a3f4-380804008a8a","Type":"ContainerStarted","Data":"220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d"} Oct 30 21:40:25.106736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.106683 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podStartSLOduration=2.10666586 podStartE2EDuration="2.10666586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:25.102475825 +0000 UTC m=+687.298166945" watchObservedRunningTime="2026-10-30 21:40:25.10666586 +0000 UTC m=+687.302356980" Oct 30 21:40:25.126851 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.126809 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podStartSLOduration=2.126797106 podStartE2EDuration="2.126797106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:25.125218466 +0000 UTC m=+687.320909599" watchObservedRunningTime="2026-10-30 21:40:25.126797106 +0000 UTC m=+687.322488225" Oct 30 21:40:25.414065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.414030 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:25.426370 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.426343 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:25.669137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.669042 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:25.669529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.669507 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:40:25.669744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.669704 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:40:25.996532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:25.996444 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:26.011971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.001279 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:26.011971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.001336 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:40:26.122259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.122234 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:26.669973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.669354 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:40:26.669973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.669420 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:40:26.959025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.958294 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:26.959025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:26.958353 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:40:27.666976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:27.664334 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:40:27.666976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:27.664435 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:40:27.959027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:27.958259 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:27.959027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:27.958328 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:40:28.668973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:28.663285 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:40:28.668973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:28.663359 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:40:28.961027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:28.957218 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:28.961027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:28.957286 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:40:29.669974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:29.669311 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:40:29.669974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:29.669380 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:40:29.959031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:29.958271 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:29.959031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:29.958327 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:40:30.438324 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.437346 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm"] Oct 30 21:40:30.454420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.454393 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm"] Oct 30 21:40:30.454748 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.454691 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.467165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.467141 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-000f64\"/\"distinct-2-w0-000f64-kserve-self-signed-certs\"" Oct 30 21:40:30.579748 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.579708 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.580026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.580004 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.580233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.580215 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgpp\" (UniqueName: \"kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.580352 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.580338 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.580504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.580491 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.580623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.580610 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.674973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.662342 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:40:30.692022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.678755 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.775522 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgpp\" (UniqueName: \"kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.775613 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.775720 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.775766 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.775933 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.775983 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.776668 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.782071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.777306 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.783021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.782996 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.793035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.787191 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.802655 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.802627 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgpp\" (UniqueName: \"kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.831279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.825508 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm\") pod \"distinct-2-w0-000f64-kserve-57b5476d6-8d2zm\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.831279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.830303 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:30.958745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.958311 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:30.958745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:30.958371 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:40:31.267359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:31.267325 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm"] Oct 30 21:40:31.270705 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:31.270677 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfa354d4_5b05_4fc6_9120_8684f0d2c742.slice/crio-fb8c8e382930277a483cd35678950c6c308dcf1e46c69348559a3a4cbeb87a4c WatchSource:0}: Error finding container fb8c8e382930277a483cd35678950c6c308dcf1e46c69348559a3a4cbeb87a4c: Status 404 returned error can't find the container with id fb8c8e382930277a483cd35678950c6c308dcf1e46c69348559a3a4cbeb87a4c Oct 30 21:40:32.108540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.108497 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:32.112375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.112353 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:32.129212 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.125504 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:40:32.220139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.219564 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" event={"ID":"bfa354d4-5b05-4fc6-9120-8684f0d2c742","Type":"ContainerStarted","Data":"8cffa7465e40afcbcab708342325b94b9a3710d5bcc8ba8d4ce2775a5e6c545c"} Oct 30 21:40:32.220139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.219611 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" event={"ID":"bfa354d4-5b05-4fc6-9120-8684f0d2c742","Type":"ContainerStarted","Data":"fb8c8e382930277a483cd35678950c6c308dcf1e46c69348559a3a4cbeb87a4c"} Oct 30 21:40:32.243798 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.242401 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podStartSLOduration=2.2423804609999998 podStartE2EDuration="2.242380461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:32.241061203 +0000 UTC m=+694.436752324" watchObservedRunningTime="2026-10-30 21:40:32.242380461 +0000 UTC m=+694.438071582" Oct 30 21:40:32.482463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.482429 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:32.482831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.482807 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:32.483935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.483914 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:40:32.830763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:32.830667 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:35.855975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:35.855252 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.894969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:35.894042 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.914976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:35.900076 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.939168 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:40:35.939129 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:40:36.123027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.119039 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.136934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.130054 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:36.136934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.130107 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.463102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.463053 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.769134 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.769034 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.769743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.769365 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.903973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.903386 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.903973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.903853 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:36.913970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.905033 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.913970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.905149 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:40:36.931391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.930623 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:36.953487 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:36.953453 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:40:39.086134 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.086102 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h"] Oct 30 21:40:39.090887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.090863 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.114804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.114777 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h"] Oct 30 21:40:39.164385 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164351 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164557 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164478 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164621 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164552 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njwg\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164621 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164598 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164634 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164738 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164770 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164800 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.164888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.164855 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.265692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.265659 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.265871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.265701 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.265871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.265745 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.265871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.265812 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.265905 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.265945 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njwg\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266025 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266205 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266063 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266205 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266116 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266212 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266307 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266508 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266486 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266580 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.266920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.266893 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.268244 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.268214 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.268433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.268416 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.278632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.278606 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njwg\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.278845 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.278816 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.416837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.416804 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:39.716043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.716013 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h"] Oct 30 21:40:39.717132 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:40:39.717108 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c75041_02c6_4326_ae01_bb651acf4497.slice/crio-90e0d429fa91de3367397817768e0bb53ff79de4a6ec507adf1b8911442dcb34 WatchSource:0}: Error finding container 90e0d429fa91de3367397817768e0bb53ff79de4a6ec507adf1b8911442dcb34: Status 404 returned error can't find the container with id 90e0d429fa91de3367397817768e0bb53ff79de4a6ec507adf1b8911442dcb34 Oct 30 21:40:39.720273 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.720240 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:39.720369 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.720307 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:39.720369 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:39.720338 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:40:40.371357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.371311 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" event={"ID":"60c75041-02c6-4326-ae01-bb651acf4497","Type":"ContainerStarted","Data":"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603"} Oct 30 21:40:40.371861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.371401 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" event={"ID":"60c75041-02c6-4326-ae01-bb651acf4497","Type":"ContainerStarted","Data":"90e0d429fa91de3367397817768e0bb53ff79de4a6ec507adf1b8911442dcb34"} Oct 30 21:40:40.398984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.398898 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podStartSLOduration=1.398883319 podStartE2EDuration="1.398883319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:40.396597086 +0000 UTC m=+702.592288206" watchObservedRunningTime="2026-10-30 21:40:40.398883319 +0000 UTC m=+702.594574503" Oct 30 21:40:40.417086 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.417059 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:40.418838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.418816 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:40.418906 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.418860 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:40.831522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.831486 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:40.877595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:40.877561 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:41.403701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:41.403675 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:40:41.417696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:41.417669 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:41.417826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:41.417712 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:42.417211 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:42.417177 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:42.417618 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:42.417243 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:43.421790 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:43.420939 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:43.421790 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:43.421127 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:44.420453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:44.420417 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:44.420804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:44.420771 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:45.428352 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:45.428319 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:45.428932 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:45.428898 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:46.417379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:46.417346 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:40:46.417550 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:46.417409 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:40:47.422207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:47.422175 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:47.422604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:47.422441 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:47.423165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:47.423148 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:40:51.734410 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:51.734331 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:40:51.842228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:51.842184 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:51.894984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:51.892027 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:40:51.989153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:51.989057 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.113226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.113175 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.391479 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.391429 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.742020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.735494 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.742020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.736131 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:40:52.743923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.743882 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.999825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.994366 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:40:52.999825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.995550 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:40:52.999825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:52.995875 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:40:53.017101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.017062 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.032281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.032237 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.382344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.376300 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.382344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.377812 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:53.477182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.471765 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:40:53.750022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.749293 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.750507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.750200 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:40:53.786421 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.774704 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:40:53.941648 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.941613 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:40:53.942189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:53.942168 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:40:54.004123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.004028 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:54.104673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.010160 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:40:54.140251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.020554 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:54.305839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.302455 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:40:54.305839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.302540 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:40:54.305839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.302619 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:40:54.305839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.302665 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:40:54.307473 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.307446 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:40:54.310778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:40:54.310750 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:05.699070 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:41:05.699033 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:41:08.318276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.318243 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x"] Oct 30 21:41:08.323855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.323834 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.338513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.336652 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x"] Oct 30 21:41:08.358488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.358456 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7"] Oct 30 21:41:08.384326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.384292 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7"] Oct 30 21:41:08.384622 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.384605 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.456494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.456464 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t"] Oct 30 21:41:08.460548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.460527 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.468837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.468813 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t"] Oct 30 21:41:08.488691 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.488659 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.488915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.488893 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgbw\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489213 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.489281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489258 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489290 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489332 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.489495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489383 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zm9j\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.489495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489415 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.489495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489477 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489505 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489529 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489569 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.489719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489604 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489634 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.489719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489704 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.490180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489820 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.490180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489883 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.490180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.489921 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591217 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591280 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.591296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591304 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591329 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591416 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.591564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591511 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591531 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgbw\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591569 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591601 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591628 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591655 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591687 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591712 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591713 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591757 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zm9j\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.591788 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591775 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591807 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591861 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591892 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.591970 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592005 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592032 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592039 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592060 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592107 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592135 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbkx\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592172 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592205 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.592295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592258 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.593041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592438 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.593041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592671 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.593041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592733 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.593041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592879 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.593041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592918 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.593041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.592928 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.593266 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.593129 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.593707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.593681 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.593907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.593889 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.594558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.594537 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.595278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.595255 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.595407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.595389 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.604353 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.604334 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.604468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.604431 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.605096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.605074 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgbw\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-m562x\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.605185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.605094 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zm9j\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.640126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.640100 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:08.694676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694630 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.694843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694706 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.694843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694786 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.694843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694816 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.695030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694873 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.695030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694899 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.695030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.694965 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.695030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.695013 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbkx\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.695231 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.695062 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.698047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.695614 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.698047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.695658 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.698047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.695909 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.698047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.695984 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.698047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.696526 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.701268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.701241 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.703784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.703757 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.716161 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.708552 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:08.716161 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.715887 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.716161 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.715906 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbkx\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.780812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.780086 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:08.805524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.793739 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn"] Oct 30 21:41:08.805524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.803724 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:08.813444 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.813410 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn"] Oct 30 21:41:08.936568 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.934533 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x"] Oct 30 21:41:08.936676 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:41:08.936578 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b88d721_c32d_487a_b10d_1a5c48e940ce.slice/crio-f70586257499b64b04e6bf3988917ffd6c35c5280b94066144a5d5f62ccfdba4 WatchSource:0}: Error finding container f70586257499b64b04e6bf3988917ffd6c35c5280b94066144a5d5f62ccfdba4: Status 404 returned error can't find the container with id f70586257499b64b04e6bf3988917ffd6c35c5280b94066144a5d5f62ccfdba4 Oct 30 21:41:08.943613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.943575 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:08.943730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.943677 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:08.943795 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:08.943737 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.001296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001270 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hqc\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001334 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001421 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001451 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001550 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001535 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001606 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001651 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001729 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001718 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.001763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.001742 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.019461 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.019386 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7"] Oct 30 21:41:09.027709 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:41:09.027645 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a3a86b_dbb8_4a28_ab1b_42b1289bd1c1.slice/crio-91744b467c4ddd9d7f9fb765915c0355ecb629fbc44308ffc5eb8abdc9bf87ba WatchSource:0}: Error finding container 91744b467c4ddd9d7f9fb765915c0355ecb629fbc44308ffc5eb8abdc9bf87ba: Status 404 returned error can't find the container with id 91744b467c4ddd9d7f9fb765915c0355ecb629fbc44308ffc5eb8abdc9bf87ba Oct 30 21:41:09.030296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.030264 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.030382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.030330 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.030382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.030366 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.104990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.104734 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hqc\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.104990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.104810 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.104990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.104865 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.104990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.104906 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.105361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.105341 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.111220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.105915 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.111220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.106824 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.111220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.107214 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.111220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.107623 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.115260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.112408 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.115260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.112540 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.115260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.112577 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.115260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.113217 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.115260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.113344 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.124325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.117245 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.124325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.119095 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.128228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.127454 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hqc\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.131736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.130832 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.170175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.165257 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:09.182987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.177827 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t"] Oct 30 21:41:09.221305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.221256 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.221571 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.221540 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.221679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.221594 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.532742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.532651 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn"] Oct 30 21:41:09.540320 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:41:09.540285 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7f4129_3d0c_4a2f_803a_344793e06133.slice/crio-c3d3fe5ccafdc290e53cab7b94e8c9feb0c6d6d050b66c077e49e42bfa97a651 WatchSource:0}: Error finding container c3d3fe5ccafdc290e53cab7b94e8c9feb0c6d6d050b66c077e49e42bfa97a651: Status 404 returned error can't find the container with id c3d3fe5ccafdc290e53cab7b94e8c9feb0c6d6d050b66c077e49e42bfa97a651 Oct 30 21:41:09.546227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.546105 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.546472 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.546408 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:09.546712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:09.546647 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:10.514875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.513969 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" event={"ID":"4b88d721-c32d-487a-b10d-1a5c48e940ce","Type":"ContainerStarted","Data":"cd7b5c2b769c4bb2c7d4edc893857d0d7a5bd81e96722154e09174253b18e7e3"} Oct 30 21:41:10.514875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.514038 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" event={"ID":"4b88d721-c32d-487a-b10d-1a5c48e940ce","Type":"ContainerStarted","Data":"f70586257499b64b04e6bf3988917ffd6c35c5280b94066144a5d5f62ccfdba4"} Oct 30 21:41:10.587142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.570290 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podStartSLOduration=2.570270552 podStartE2EDuration="2.570270552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:10.567438101 +0000 UTC m=+732.763129221" watchObservedRunningTime="2026-10-30 21:41:10.570270552 +0000 UTC m=+732.765961672" Oct 30 21:41:10.599277 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.580229 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" event={"ID":"cc153255-a9df-48c9-9f5a-dc57f2834c52","Type":"ContainerStarted","Data":"b6a27811363b7287c9cfde2539e27f621e0604e32b02b3422df2577a83eff5c4"} Oct 30 21:41:10.599594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.599564 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" event={"ID":"cc153255-a9df-48c9-9f5a-dc57f2834c52","Type":"ContainerStarted","Data":"98a37102a58a5bd0b31561ff2b6b6a171e9d339887872db4bc3104e13e69ecd5"} Oct 30 21:41:10.619237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.619195 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" event={"ID":"0a7f4129-3d0c-4a2f-803a-344793e06133","Type":"ContainerStarted","Data":"4b1dfe3ef0e4725ded2b69791223c16deea522158aa02298579f0bdd49bcf018"} Oct 30 21:41:10.620107 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.620077 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" event={"ID":"0a7f4129-3d0c-4a2f-803a-344793e06133","Type":"ContainerStarted","Data":"c3d3fe5ccafdc290e53cab7b94e8c9feb0c6d6d050b66c077e49e42bfa97a651"} Oct 30 21:41:10.627227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.626461 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podStartSLOduration=2.626442955 podStartE2EDuration="2.626442955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:10.60957702 +0000 UTC m=+732.805268151" watchObservedRunningTime="2026-10-30 21:41:10.626442955 +0000 UTC m=+732.822134078" Oct 30 21:41:10.633215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.633187 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" event={"ID":"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1","Type":"ContainerStarted","Data":"bae25150204a2ea3e5f122cdbe596da7b30af0618087b7c6d276c8d672835817"} Oct 30 21:41:10.633416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.633401 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" event={"ID":"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1","Type":"ContainerStarted","Data":"91744b467c4ddd9d7f9fb765915c0355ecb629fbc44308ffc5eb8abdc9bf87ba"} Oct 30 21:41:10.669759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.647012 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:10.695971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.693706 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podStartSLOduration=2.6936831039999998 podStartE2EDuration="2.693683104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:10.670588549 +0000 UTC m=+732.866279670" watchObservedRunningTime="2026-10-30 21:41:10.693683104 +0000 UTC m=+732.889374215" Oct 30 21:41:10.718859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.718824 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:10.719292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.719263 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.719484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.719456 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:10.823032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.783998 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.823032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.784055 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:10.823032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.787303 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.823032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.787346 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:10.823032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.787496 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:10.823032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:10.822482 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podStartSLOduration=2.822458996 podStartE2EDuration="2.822458996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:10.717438206 +0000 UTC m=+732.913129326" watchObservedRunningTime="2026-10-30 21:41:10.822458996 +0000 UTC m=+733.018150116" Oct 30 21:41:11.167140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.167097 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:11.169631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.169604 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.169861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.169826 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:11.655636 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.655598 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.659713 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.659659 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:11.897121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.897077 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.897369 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.897343 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:11.909419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.905198 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.909419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.905247 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:11.944404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.944361 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:11.991095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:11.991051 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.057090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.057037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.134101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.134042 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.226033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.207211 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.226033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.207274 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:12.323581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.323534 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.324183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.324156 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:12.394112 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.394062 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.641452 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.641409 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.641794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.641760 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:12.723976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.709559 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.730858 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.730815 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:12.783364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.783321 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.783606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:12.783577 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:13.001154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.001050 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.002069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.002047 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:13.154854 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.154801 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.169108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.169060 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:13.221004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.220349 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.221004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.220405 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:13.326038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.325920 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.350745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.350640 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:13.350942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.335488 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:13.414996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.396003 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.427370 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.404586 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:13.461979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.458061 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:13.549994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.549494 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:13.651107 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.651074 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.651416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.651371 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:13.722018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.721982 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.723501 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.723465 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:13.789908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.786420 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.789908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.786480 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:13.969028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:13.966305 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.067994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.003286 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.067994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.036407 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:14.169102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.169036 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:14.183316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.183276 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.183502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.183336 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:14.356768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.356659 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:14.451499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.444907 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:14.476973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.476765 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:14.666972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.666200 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.666972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.666267 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:14.728068 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.728025 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.735971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.735921 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.736189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.736167 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:14.782684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.782487 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.783085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.783054 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:14.792880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.792857 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:14.984214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.984109 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.997227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:14.989757 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:15.168657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.168622 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.168925 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.168880 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:15.363133 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.363031 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:15.483154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.483093 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:15.651980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.651109 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.651980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.651345 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:15.729970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.724611 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.729970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.724699 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:15.786790 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.786755 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.790990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.787054 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:15.994438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:15.994294 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:41:16.033064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.033021 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.048288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.048243 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.170972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.110129 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.195870 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.188861 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:16.196582 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.195018 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.196801 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.196772 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:16.356326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.355543 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.364870 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.364269 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:16.414656 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.414593 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.553013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.552939 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:16.646399 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.646362 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.654976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.646638 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:16.735446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.734360 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.735446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.734419 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:16.795636 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.795237 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 21:41:16.811100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.811067 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.830183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.820041 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.830891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:16.830858 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:17.029031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.028022 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:41:17.055079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.055028 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:17.056030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.056006 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:41:17.159882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.159841 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:17.162154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.161137 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:41:17.167593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.167563 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.167717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.167616 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:17.168913 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.168893 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:17.198085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.198045 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:17.198676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.198648 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:17.205998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.205975 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:17.206212 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.206197 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:41:17.534431 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.534400 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:41:17.535915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.535882 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:17.536136 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.536099 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:17.537175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.536360 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:17.572362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.555177 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:17.572362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.561228 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:17.617405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.617377 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:17.761980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.729131 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.761980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.729190 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:17.843020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.831941 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:17.843020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.836119 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:17.915987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.907662 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.915987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.907716 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:41:17.915987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.907909 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.951982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:17.907937 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:18.164075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.164025 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:18.209176 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.209136 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:18.209811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.209780 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:18.245645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.245617 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:18.301250 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.301217 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:18.301525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.301483 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:18.336694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.336660 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:18.458028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.446557 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:18.617092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.616976 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:41:18.640687 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.640654 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:41:18.642321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.641902 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:41:18.724404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.724318 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:18.781511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.781477 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:18.781873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.781822 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:41:18.928987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:18.928866 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:41:19.181020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.180016 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:19.181020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.180073 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:19.744051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.744012 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:19.744344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.744311 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:19.798836 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.798792 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:19.801588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.801564 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:19.806353 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:19.806335 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:41:20.011517 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.011200 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:20.012188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.012166 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:20.013161 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.013139 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:41:20.176075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.176042 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:41:20.176326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.176299 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:41:20.362396 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.362310 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:20.365115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.365093 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:20.389280 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:20.389245 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:41:21.237903 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:21.237872 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:21.238814 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:21.238787 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:21.251623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:21.250675 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:41:23.922029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:23.845451 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl"] Oct 30 21:41:24.212451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.212330 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.226973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.212940 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf"] Oct 30 21:41:24.226973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.213328 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.250968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.242066 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.299975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.294180 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324530 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324583 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324630 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324658 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324683 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324710 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324757 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324791 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324819 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.324973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324857 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdfn\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.325588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.324989 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.325588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.325017 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.384114 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.343464 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.384114 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.381918 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.414992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.395097 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.414992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.395167 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.414992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.395229 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.414992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.395270 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:24.414992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.395326 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.414992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.395443 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkc6l\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:24.609210 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.609115 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.611291 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.611510 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.643652 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.643847 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.650756 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.046392 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.046488 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.046541 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.046974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.651431 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.047928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.830210 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 21:41:25.047928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:24.988199 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl"] Oct 30 21:41:25.047928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.047153 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf"] Oct 30 21:41:25.047928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.047201 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9"] Oct 30 21:41:25.181972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.181824 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.206970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.182180 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.206970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.182251 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 21:41:25.206970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.182756 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.206970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.183379 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkc6l\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.206970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.187176 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.268972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.268035 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.268972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.268608 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.269220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.269186 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.281971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.269517 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.282836 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.283442 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.283514 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.284286 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.197824 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293192 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293238 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293265 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293318 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293383 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293439 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293466 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.307018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.293519 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdfn\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.358973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.355296 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.362971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.362465 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.368535 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.368358 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.397876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.376971 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.397876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.392522 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.530159 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.489253 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.530159 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.489971 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.542556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.542492 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.626705 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.626604 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.693005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.632146 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.693005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.632812 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkc6l\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.696969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.693392 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk"] Oct 30 21:41:25.696969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.694121 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.749974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.723732 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdfn\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.751847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.751811 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.752186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.752162 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.752377 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.752357 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.752528 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.752512 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.752640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.752624 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.752759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.752745 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.752862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.752847 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.753020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.753000 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.753165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.753149 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766m2\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.761981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.754434 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.798502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.798471 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9"] Oct 30 21:41:25.798860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.798793 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:25.799309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.799252 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:25.799309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.799300 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:25.799309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.799321 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk"] Oct 30 21:41:25.799593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.799371 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:25.799691 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.798699 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:25.841727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.841691 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:25.884076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.874572 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766m2\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.884493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.884464 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.924894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.924857 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.964182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.964149 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.965094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.965069 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.965787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.965767 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.966009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.965986 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.980763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.980738 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.982309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.962585 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:25.982309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.982278 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.982309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.926576 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.982309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.980262 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.982309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.890666 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.982728 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.966326 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:25.991456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:25.991357 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:26.034143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.034102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:41:26.066975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.042251 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:26.066975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.060744 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.109896 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110013 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110113 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110156 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110197 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4g9z\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110339 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110367 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110498 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.112973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.110561 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.148022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.116394 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:26.197018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.174207 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:26.197018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.182623 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766m2\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:26.197579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.197547 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.197752 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.198810 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.219171 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.220795 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.220915 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.220994 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.221051 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.221122 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.221189 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.221226 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.221262 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4g9z\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.221923 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.222621 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.219570 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.222891 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.223514 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.226738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.224436 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.239885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.239774 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:26.340067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.313058 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.340067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.313795 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:41:26.340067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.315992 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.359923 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.360532 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.369486 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.369743 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.370102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.370362 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.387413 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.388079 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:41:26.388552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.388377 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:26.459789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.459730 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4g9z\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.531972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.487475 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.531972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.524322 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:26.718871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.709422 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.718871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.712751 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:26.718871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.715488 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:26.783977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.780186 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 21:41:26.907060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:26.892391 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:27.043142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.043042 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.248093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.248046 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.346700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.346602 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.406020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.373819 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:27.406020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.374148 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:27.406020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.374406 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:41:27.406020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.393527 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.610035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.580074 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:27.720130 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.720081 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:27.836976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:27.833338 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.147041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.074667 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.270291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.249060 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:28.758541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.755287 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:28.833613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.771625 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:28.833613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.771937 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.833613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.773174 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:41:28.833613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.815921 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:28.833613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.816501 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:28.896163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:28.877459 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:41:29.061072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.060799 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.221814 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.221781 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:29.222118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.222093 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.669630 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:41:29.650457 6808 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.608s" Oct 30 21:41:29.722390 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.722357 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9"] Oct 30 21:41:29.722730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.722711 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl"] Oct 30 21:41:29.758834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.747098 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk"] Oct 30 21:41:29.794078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.794040 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:41:29.795048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.794302 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.807973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.794334 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:41:29.819371 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.795277 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:29.835985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.820413 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.868014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.824356 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.948359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:29.948302 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:41:30.085842 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.073971 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:30.102229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.102172 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:41:30.118974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.111580 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.204451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.127691 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.205057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.152420 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:30.205255 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.205212 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.298968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.286903 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.326778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.288402 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.327109 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.289746 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.327518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.327466 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.327798 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.327746 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.328209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.328079 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.328774 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.328743 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.329424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.328679 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.329767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.329741 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.436972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.417607 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.763979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.693683 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.810237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.810190 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:30.810966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.741431 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:41:30.811158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.811141 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf"] Oct 30 21:41:30.811302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.811282 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" event={"ID":"1fce4081-a70c-4873-8ef6-de2a95d44def","Type":"ContainerStarted","Data":"032e1aa70d03ffc256d94ed3d282c004c9bf8f1e739402faab2255112881c424"} Oct 30 21:41:30.828976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.750296 6808 patch_prober.go:29] interesting pod/thanos-querier-6f995ff688-xs5n7 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:30.829225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.829196 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-6f995ff688-xs5n7" podUID="b4e22dd8-f90b-4512-9ade-3f3ea3329946" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.860391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.816860 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" event={"ID":"72c61b81-7a11-4f8d-852d-0de74e31d752","Type":"ContainerStarted","Data":"9a164e25137797877fa9ca83ca2bdac66c389798873eb251706db56438fe23e9"} Oct 30 21:41:30.860618 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.826442 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.861083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.807939 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.861447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.840783 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.901455 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:30.900219 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.007022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.006153 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.075976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.064964 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" event={"ID":"79d92887-ea5d-4ced-880e-de31ce124c20","Type":"ContainerStarted","Data":"eaa7baf866ca1a3c50a13f80d06660540d79a59c9de577c89f8804f4cb66c8d1"} Oct 30 21:41:31.105101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.105049 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.137081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.137032 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.170021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.169978 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.196214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.182940 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" event={"ID":"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea","Type":"ContainerStarted","Data":"8aee7a7b29984a3f16384a86c9db5ea7f83be09e23f77fd84bd77c3ef0e41e05"} Oct 30 21:41:31.212983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.206032 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.212983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.206402 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.304027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.303776 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:41:31.608010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.605859 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:31.689003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.666283 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.689003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.670233 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:31.859480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.845778 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.862008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.861975 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.862629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.862231 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:41:31.899029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.890622 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.914016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:31.904102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:32.227099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.227058 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:32.232502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.232466 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:32.255046 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.251983 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:32.323073 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.317043 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:32.403973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.389225 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:32.477871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.476699 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:32.477871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.476757 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" event={"ID":"72c61b81-7a11-4f8d-852d-0de74e31d752","Type":"ContainerStarted","Data":"ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e"} Oct 30 21:41:32.491881 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.485085 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" event={"ID":"79d92887-ea5d-4ced-880e-de31ce124c20","Type":"ContainerStarted","Data":"8af2002b72a3a54762ed329dec53fff462ff387fbb9b04a2efcab376c00cb565"} Oct 30 21:41:32.643798 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.643755 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:32.756718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.756679 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:32.909480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.909367 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:32.911033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.911003 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:32.911300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.911208 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:32.912870 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.911448 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:41:32.938678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.931139 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:32.940922 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.940877 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podStartSLOduration=8.940847573 podStartE2EDuration="8.940847573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:32.770622019 +0000 UTC m=+754.966313151" watchObservedRunningTime="2026-10-30 21:41:32.940847573 +0000 UTC m=+755.136538694" Oct 30 21:41:32.979484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:32.979442 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:33.080406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.080334 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podStartSLOduration=10.080310694 podStartE2EDuration="10.080310694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:32.939169524 +0000 UTC m=+755.134860686" watchObservedRunningTime="2026-10-30 21:41:33.080310694 +0000 UTC m=+755.276001814" Oct 30 21:41:33.151627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.151574 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:33.151809 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.151641 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:33.164365 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.164286 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:33.182386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.182351 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:41:33.221569 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.221534 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:33.227840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.227328 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:33.239500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.239109 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.239500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.239168 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:33.516903 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.516817 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" event={"ID":"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea","Type":"ContainerStarted","Data":"8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9"} Oct 30 21:41:33.521568 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.521538 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" event={"ID":"1fce4081-a70c-4873-8ef6-de2a95d44def","Type":"ContainerStarted","Data":"1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33"} Oct 30 21:41:33.599808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.599744 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podStartSLOduration=9.599722318 podStartE2EDuration="9.599722318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.565594774 +0000 UTC m=+755.761285894" watchObservedRunningTime="2026-10-30 21:41:33.599722318 +0000 UTC m=+755.795413437" Oct 30 21:41:33.740987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.740419 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.740987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:33.740479 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:34.148971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.148150 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:34.168979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.165068 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.188972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.165159 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:34.212110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.212072 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.212358 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.212331 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:34.260217 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.258761 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:34.260676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.258851 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.260850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.260829 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:34.719973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.719365 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.719973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:34.719442 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:35.140690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.132327 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.140690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.132387 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:35.232697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.224923 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.232697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.225014 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:35.249018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.248856 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.249018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.248917 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:35.741452 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.741405 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.762978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.758556 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.762978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.758627 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:35.824397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.824339 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.846027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.845204 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.875520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.874458 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:41:35.885001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.876207 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.889970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:35.885223 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.085271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.085162 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.156473 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.156436 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.156719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.156689 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:36.214276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.214240 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.214544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.214502 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:36.240977 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:41:36.230854 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:41:36.262439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.260088 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.262439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.260163 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:36.385970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.383331 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.545975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.530051 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.719625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.719504 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.719911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.719862 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:36.762134 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.760398 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:36.762684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.762641 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:36.764344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.764319 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.874974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.857623 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podStartSLOduration=13.857598348 podStartE2EDuration="13.857598348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.632097548 +0000 UTC m=+755.827788673" watchObservedRunningTime="2026-10-30 21:41:36.857598348 +0000 UTC m=+759.053289468" Oct 30 21:41:36.903974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.878269 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.906291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.906258 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:41:36.907252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.907222 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.923978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.907860 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.924731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.924704 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:36.924965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.924934 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:41:36.927361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.927333 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.933972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.933676 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:37.543097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:36.985891 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.561999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.090513 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.562922 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.563046 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.130405 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.563307 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.470125 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.563390 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.331881 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:37.595275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.395471 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.623003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.616997 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:37.623003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.618429 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.623003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.618486 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:37.623003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.619544 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.623551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.623519 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:37.859360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.809395 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:37.859360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.824537 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.888319 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.875912 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 21:41:37.892263 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.892238 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.892464 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.892440 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:37.936076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.936032 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:41:37.943104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.943039 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:37.968133 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:37.968096 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.309974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.303675 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.309974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.303747 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:38.353019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.349400 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.353377 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.353327 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:38.377148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.375218 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.485143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.485101 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:38.558533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.558485 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.564129 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.564028 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.564455 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.564431 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:38.690974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.686056 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.690974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.686307 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:38.871324 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.871227 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.895973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.887182 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:38.895973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.892145 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:38.937071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.937032 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.979088 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.979041 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.980586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:38.980548 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.023972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.023922 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.031267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.031214 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:39.146025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.141872 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.146025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.141928 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:39.220772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.219309 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.220772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.219369 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:39.257771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.257737 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.258045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.258016 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:39.559881 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.559721 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.577785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.576131 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.579169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.577326 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.705514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.705472 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.706069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.706033 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.737562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.737526 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.737820 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.737791 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:39.886091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.886050 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:41:39.887020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.886998 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:39.907817 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.898087 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.945970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:39.943379 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.024970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.023117 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.024970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.024070 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:41:40.176063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.140648 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.176063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.140703 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:40.239831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.203062 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.239831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.203116 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:40.239831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.229215 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:40.364859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.302287 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.365479 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.365440 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:40.581073 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.577561 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.585234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.585022 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:40.717971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.714084 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.717971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.715275 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:40.740971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.735589 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.740971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.735761 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:40.916972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.912050 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.916972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.913193 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.945487 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:40.945445 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:41:41.205162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.204675 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.205162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.204735 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:41.248510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.246086 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.248510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.246143 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:41.248510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.246767 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:41.248510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.247740 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 30 21:41:41.248510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.247823 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:41.248899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.248812 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.248899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.248846 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:41.249227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.248899 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.271506 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.271062 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:41.282678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.282625 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:41:41.307970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.292037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.335671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.335643 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:41:41.548099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.547966 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.621194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.590225 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.652413 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.614028 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.653116 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.639631 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:41:41.654277 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.640381 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:41:41.735100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.735053 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.736193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.735378 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:41.736584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.735499 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:41.753963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.753915 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.754193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.754164 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:41.862188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:41.860976 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:42.169984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.168126 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.169984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.168206 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:42.218523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.216167 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.218523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.216247 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:41:42.272619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.272583 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.272891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.272859 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:42.307620 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.307576 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:42.681115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.679662 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:42.698673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.697565 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:42.730357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.730319 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.730701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.730647 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:42.735984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.735938 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:41:42.736768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.736748 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:42.955028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:42.950221 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:43.025091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.025045 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:43.190972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.190777 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.190972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.190837 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:43.205002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.201487 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:41:43.227283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.227208 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:41:43.241604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.241563 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.241839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.241814 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:41:43.344095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.344064 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:43.522426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.510561 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:43.736970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.732745 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.736970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:43.732802 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:41:44.125541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.125507 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.126329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.125813 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:41:44.239979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.239314 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:44.239979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.239393 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:44.390023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.381566 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:44.390023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.386232 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:41:44.774300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.773291 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:44.799266 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:44.799237 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:41:45.021278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:45.021243 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:45.023057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:45.023030 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:41:45.504539 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:45.504032 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:45.505581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:45.505558 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:41:53.945736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:53.945698 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n"] Oct 30 21:41:53.949898 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:53.949877 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:53.970941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:53.970498 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n"] Oct 30 21:41:54.003412 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.003380 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s"] Oct 30 21:41:54.008234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.008214 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.022685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.022656 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s"] Oct 30 21:41:54.033722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.033696 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.033849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.033770 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.033915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.033859 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.033984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.033922 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.034053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.034027 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.034093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.034073 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.034141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.034126 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.034278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.034262 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbw4\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.034422 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.034406 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135379 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135414 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.135564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135456 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.135564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135496 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135579 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135620 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.135707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135659 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135719 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135756 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135801 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.135861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135838 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.136128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135877 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.136128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135943 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbw4\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.136128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.135999 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.136128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136003 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.136128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136021 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.136128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136088 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.136356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136219 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.136356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136245 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.136356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136336 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.136485 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136396 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.136485 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136431 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.136586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.136484 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.138060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.138037 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.138463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.138440 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.151308 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.151281 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.151627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.151603 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbw4\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.237942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.237855 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.237942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.237921 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238033 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238094 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238173 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238232 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238309 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238338 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238352 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238440 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238536 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238722 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.238888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.238861 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.239176 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.239154 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.240904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.240878 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.241394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.241374 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.250767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.250739 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.251041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.251022 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.264422 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.264400 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:54.333905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.333400 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:54.384300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.384267 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc"] Oct 30 21:41:54.389580 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.388944 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.413975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.409212 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc"] Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597314 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597372 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597408 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2qw\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597487 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597536 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597576 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597653 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597679 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.599058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.597829 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.699558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699513 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.699738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699581 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.699738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699618 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2qw\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.699865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699742 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.699865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699796 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.699865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699843 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.700050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699924 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.700050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.699997 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.700203 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.700184 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.700405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.700380 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.700520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.700499 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.700818 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.700777 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.701421 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.701378 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.701603 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.701577 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.703049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.703024 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.704002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.703976 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.712699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.712664 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.713189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.713165 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2qw\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.767739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.767700 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n"] Oct 30 21:41:54.768464 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:41:54.768430 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930b3f3b_88d2_4122_8da4_2fc04f449713.slice/crio-39eedf86981bdb6298f2eb0e4ef114e345b3be34ed7b4095ff4b9dc41bcec828 WatchSource:0}: Error finding container 39eedf86981bdb6298f2eb0e4ef114e345b3be34ed7b4095ff4b9dc41bcec828: Status 404 returned error can't find the container with id 39eedf86981bdb6298f2eb0e4ef114e345b3be34ed7b4095ff4b9dc41bcec828 Oct 30 21:41:54.771656 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.771617 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:54.771744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.771700 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:54.771800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.771763 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:54.799680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.799656 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:54.854530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.854503 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s"] Oct 30 21:41:54.854711 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:41:54.854640 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc26646_e9c2_4096_90ad_4b0df42ba0e1.slice/crio-1adddbebec7387f8556598f6b4aef3074d288fc1c50915b4e1013b38d6e6dc10 WatchSource:0}: Error finding container 1adddbebec7387f8556598f6b4aef3074d288fc1c50915b4e1013b38d6e6dc10: Status 404 returned error can't find the container with id 1adddbebec7387f8556598f6b4aef3074d288fc1c50915b4e1013b38d6e6dc10 Oct 30 21:41:54.859354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.859317 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:54.859446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.859403 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:54.859506 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.859449 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:54.927986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.927937 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" event={"ID":"930b3f3b-88d2-4122-8da4-2fc04f449713","Type":"ContainerStarted","Data":"48151609b60b2599d79a060382b22d7bb925f54490013abeb533fefb3dca66b2"} Oct 30 21:41:54.928197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.928175 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" event={"ID":"930b3f3b-88d2-4122-8da4-2fc04f449713","Type":"ContainerStarted","Data":"39eedf86981bdb6298f2eb0e4ef114e345b3be34ed7b4095ff4b9dc41bcec828"} Oct 30 21:41:54.930158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.930126 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" event={"ID":"ebc26646-e9c2-4096-90ad-4b0df42ba0e1","Type":"ContainerStarted","Data":"1adddbebec7387f8556598f6b4aef3074d288fc1c50915b4e1013b38d6e6dc10"} Oct 30 21:41:54.965674 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:54.960881 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podStartSLOduration=1.960860652 podStartE2EDuration="1.960860652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:54.957976614 +0000 UTC m=+777.153667735" watchObservedRunningTime="2026-10-30 21:41:54.960860652 +0000 UTC m=+777.156551777" Oct 30 21:41:55.127011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.126960 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc"] Oct 30 21:41:55.129799 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.129772 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:55.129931 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.129915 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:55.130016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.129998 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:41:55.269987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.267320 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:41:55.285970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.280116 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:55.285970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.280181 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:55.937819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.937782 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" event={"ID":"d367b0db-fb5d-4255-ac52-7a6c07d48275","Type":"ContainerStarted","Data":"113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273"} Oct 30 21:41:55.938043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.937844 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" event={"ID":"d367b0db-fb5d-4255-ac52-7a6c07d48275","Type":"ContainerStarted","Data":"acb7618ea2094ede0c4bc0c3e1989396b6ee9fc6ce18fc32fae7ad57de6030da"} Oct 30 21:41:55.939816 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.939789 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" event={"ID":"ebc26646-e9c2-4096-90ad-4b0df42ba0e1","Type":"ContainerStarted","Data":"5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb"} Oct 30 21:41:55.980382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:55.980324 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podStartSLOduration=1.9802924549999998 podStartE2EDuration="1.980292455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.980173268 +0000 UTC m=+778.175864388" watchObservedRunningTime="2026-10-30 21:41:55.980292455 +0000 UTC m=+778.175983575" Oct 30 21:41:56.005004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.004937 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podStartSLOduration=3.004924196 podStartE2EDuration="3.004924196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.0028123 +0000 UTC m=+778.198503422" watchObservedRunningTime="2026-10-30 21:41:56.004924196 +0000 UTC m=+778.200615314" Oct 30 21:41:56.267034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.266432 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:56.267034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.266500 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:56.346092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.346062 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:41:56.350239 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.350204 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:56.350509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.350482 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:56.804440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.804400 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:41:56.811972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.810599 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:41:56.811972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:56.810656 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:41:57.267142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.267108 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:57.290182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.290138 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:57.353112 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.353064 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:57.353367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.353342 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:57.726301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.726249 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:57.822618 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.822578 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:41:57.840117 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.840067 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:41:57.896974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:57.896052 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:58.073053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.046118 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:58.292974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.269792 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:58.294827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.294791 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:58.402972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.386617 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:58.402972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.386712 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:58.683021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.680083 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:58.683021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.680466 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:41:58.683021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.682720 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:41:58.743973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.738037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:58.743973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.738427 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:58.743973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.739103 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:41:58.811023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.809011 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:41:58.811023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.809088 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:41:58.906979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.905024 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:58.906979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:58.905652 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:41:59.173974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.173612 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:41:59.319627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.319588 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:59.320222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.320191 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:59.344074 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.342939 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:59.344074 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.343013 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:59.344074 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.343086 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:59.376440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.375036 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:59.682912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.681747 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:59.684969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.683524 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:41:59.696785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.696752 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:59.712739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.698253 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:41:59.789972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.752625 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:59.789972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.753115 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:59.789972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.761350 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:41:59.912980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.912524 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:41:59.912980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.912580 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:41:59.954120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:41:59.924893 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:00.281254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.281172 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:00.285097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.285068 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:00.294059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.294021 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:00.339274 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.338589 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:00.339274 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.338641 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:00.659026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.613965 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:00.699970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.697039 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:00.754424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.754379 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:00.764828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.754741 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:00.813110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.813073 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:00.813966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.813921 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:00.925532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.925438 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:00.990751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:00.990015 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.017118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.004029 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.035077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.035033 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.269767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.269297 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:01.269767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.269353 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:01.338512 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.338480 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:01.338821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.338794 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.340037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.339996 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:01.660098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.660052 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.661761 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.661729 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:42:01.700974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.698632 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.764715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.764678 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.766078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.765999 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:01.850709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.850038 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 30 21:42:01.850709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.850605 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:01.850709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.850642 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:01.954035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:01.952078 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.016989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.009462 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:42:02.016989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.010144 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.070016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.030167 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.070016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.037040 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.332024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.331199 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 21:42:02.332024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.331900 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.332024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.331961 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:02.335970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.335274 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.335970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.335330 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:02.359975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.345000 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:02.359975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.345099 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:42:02.359975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.354214 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:02.359975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.354777 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:42:02.677979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.669751 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.700101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.700054 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:42:02.791977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.776090 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.897152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.794602 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.902289 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.819475 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.902608 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.902570 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:02.991166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:02.974056 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.017357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.017310 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.044192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.018646 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.105746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.073115 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:03.107270 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.073190 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.119219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.119187 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:42:03.119519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.119502 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:42:03.267657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.267570 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.267972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.267923 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:03.351435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.340061 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.352066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.352023 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:03.374591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.362020 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.569175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.569070 6808 patch_prober.go:29] interesting pod/metrics-server-bdd5667d8-qdt5r container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:03.569495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.569450 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-bdd5667d8-qdt5r" podUID="0328b8ce-272d-4232-8681-ef9e0d3a5274" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.731312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.731278 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:03.841214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.769599 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:42:03.841392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.795755 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.856772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.856734 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:03.857004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.856979 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.857119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.857026 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:03.857542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.857513 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.925157 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:03.925113 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:04.047969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.041393 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.047969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.046683 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:42:04.112757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.112651 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.115980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.115926 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.150979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.116584 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.254349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.254316 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:04.452075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.265332 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.452701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.452653 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:04.453288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.401276 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.453779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.402333 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.453964 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.453922 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:04.826312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.826219 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.827820 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.827092 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.853252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.849669 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 21:42:04.853252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.849770 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.866025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.863875 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.866025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.864450 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:42:04.866025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.865537 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.866025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.865574 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:04.935613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.935560 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.961596 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.961561 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:04.993177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:04.993138 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.097027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.081642 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.143002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.126409 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.143002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.127535 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.143002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.127796 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.345974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.344338 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.345974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.344401 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:05.365486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.362462 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.365486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.362516 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:05.459893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.459370 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:05.801973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.801315 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.801973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.801378 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:05.828978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.828029 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.828978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.828451 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:05.876978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.871081 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.876978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.871456 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:42:05.876978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.872007 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:05.876978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.872044 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.966976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.942173 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.967593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:05.945092 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.135056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.126084 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.135056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.127101 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.135056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.127885 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.135056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.128159 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:06.248029 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:42:06.234483 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:42:06.265326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.265288 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.265596 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.265566 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:06.344523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.344490 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.344806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.344767 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:06.414173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.414073 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.464167 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.464114 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.765141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.765034 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 21:42:06.803670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.803633 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.803942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.803911 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:06.831100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.831054 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.874164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.874102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.874844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.874426 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:42:06.970082 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.970037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.971293 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.971260 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.971894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:06.971874 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:07.163647 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.163593 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.164263 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.163870 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.164814 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.163884 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:07.176802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.163915 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.298861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.298239 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:07.298861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.298307 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.299252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.299226 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:07.299402 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.299381 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:07.337426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.337390 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:07.337719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.337678 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:07.466151 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.466051 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.767403 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.767090 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.826657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.803565 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:07.826657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.803621 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:07.861975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.858242 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:42:07.878123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.878051 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.879071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.878992 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:42:07.975995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.972733 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:07.976367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:07.976333 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:08.176895 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.167178 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:08.176895 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.175318 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:08.220097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.200058 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.220097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.201912 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.272980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.265265 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:08.361879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.351102 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:08.362249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.362198 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:08.382143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.382100 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:08.467181 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.467068 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.803554 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.803447 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:08.803830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.803793 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:08.866977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.862109 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.883432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.883386 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.884077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.883686 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.976975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.973102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.992988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:08.989052 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.171375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.171331 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.180076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.180034 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.201118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.201066 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.204032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.203994 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.283710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.283680 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:42:09.303602 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.303564 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:09.303911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.303878 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:09.345985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.336274 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:42:09.346898 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.346869 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:09.347128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.347097 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:09.477339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.477239 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:09.809025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.799754 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:42:09.845219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.845167 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:09.845511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.845476 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:09.865108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.865064 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:42:09.890584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.890535 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.891189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.891155 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.993982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.992018 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:09.993982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:09.992206 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.175975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.175034 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.202169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.202126 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.202743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.202709 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.229975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.209326 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:42:10.293971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.293313 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:10.293971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.293374 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:10.432021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.394651 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:10.451141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.451093 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:10.522975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.485143 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:10.825028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.820505 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:10.825028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.820566 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:10.866862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.866815 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.924973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.906692 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.924973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:10.907074 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:42:11.005169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.002070 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.005169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.003750 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:11.031077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.031034 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.179913 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.179863 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.224041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.222742 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.224041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.223133 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:11.224041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.223675 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.281408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.281370 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:11.281699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.281665 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:11.340475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.340412 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:11.340976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.340736 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:11.495251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.495121 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.766022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.762465 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:42:11.853975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.802920 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 21:42:11.912254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.912215 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:42:11.916713 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.916684 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:11.916991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.916965 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:11.917409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.917390 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.919641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:11.919600 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:12.328981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.275695 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:12.389855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.280434 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:12.390869 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.280497 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:12.401378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.280521 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:12.402704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.331028 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 21:42:12.403105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.346067 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:12.403271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.403244 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:12.403674 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.365728 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:42:12.410127 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.410102 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:12.424992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.388329 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:12.425305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.425272 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:12.514148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.514099 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:12.803604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.803516 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:12.814809 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:12.814764 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:13.076449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.076329 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:13.078666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.078629 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:13.079192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.078874 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:42:13.146940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.140694 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:42:13.226179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.226149 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:42:13.274374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.274338 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:13.559547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.559420 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:13.559793 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.395558 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:42:13.560191 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.275025 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:42:13.560433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.560420 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:13.561631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.430134 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:13.561810 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.561789 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:13.562433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.398984 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:13.710969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.707435 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:13.837030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.819046 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:13.837030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.830334 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:13.837030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.830389 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:13.974736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.974709 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:42:13.975226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:13.975210 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:14.067887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.067855 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:42:14.265545 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.265504 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:42:14.265745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.265566 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:42:14.350974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.340330 6808 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:14.350974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.340397 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:42:14.800706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.800665 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:14.800922 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:14.800718 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 21:42:16.013558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:16.013333 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:42:16.019977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:16.016094 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:42:16.126207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:16.126172 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:42:16.135682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:16.135652 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:42:16.322042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:16.318109 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:16.322042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:16.318192 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:17.016783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:17.016756 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:42:17.021265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:17.021202 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:42:21.228028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.227993 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7351-e966c7"] Oct 30 21:42:21.232215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.232193 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:21.237125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.237100 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-e966c7"] Oct 30 21:42:21.238838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.238814 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l4j2n\"" Oct 30 21:42:21.288933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.288896 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8dm\" (UniqueName: \"kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm\") pod \"test-curl-7351-e966c7\" (UID: \"fc63da54-3e43-43a7-8b8a-72b16ca11284\") " pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:21.392975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.392485 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8dm\" (UniqueName: \"kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm\") pod \"test-curl-7351-e966c7\" (UID: \"fc63da54-3e43-43a7-8b8a-72b16ca11284\") " pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:21.433063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.430420 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8dm\" (UniqueName: \"kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm\") pod \"test-curl-7351-e966c7\" (UID: \"fc63da54-3e43-43a7-8b8a-72b16ca11284\") " pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:21.547354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.547255 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:21.827786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.827756 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-e966c7"] Oct 30 21:42:21.828881 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:42:21.828852 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc63da54_3e43_43a7_8b8a_72b16ca11284.slice/crio-3a565c3d5eabab51ac76af003fa1c77a060f88b10a8bef27970e271f78b174ab WatchSource:0}: Error finding container 3a565c3d5eabab51ac76af003fa1c77a060f88b10a8bef27970e271f78b174ab: Status 404 returned error can't find the container with id 3a565c3d5eabab51ac76af003fa1c77a060f88b10a8bef27970e271f78b174ab Oct 30 21:42:21.830672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:21.830657 6808 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:42:22.054155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:22.054112 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-e966c7" event={"ID":"fc63da54-3e43-43a7-8b8a-72b16ca11284","Type":"ContainerStarted","Data":"3a565c3d5eabab51ac76af003fa1c77a060f88b10a8bef27970e271f78b174ab"} Oct 30 21:42:25.104365 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:25.104330 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-e966c7" event={"ID":"fc63da54-3e43-43a7-8b8a-72b16ca11284","Type":"ContainerStarted","Data":"c418c5f0f93c77d264c696a2214a652a1e1b633a17e0856044e1f94232aeff46"} Oct 30 21:42:25.121776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:25.121726 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7351-e966c7" podStartSLOduration=1.6271084569999998 podStartE2EDuration="4.121713741s" totalImagesPullingTime="2.494605284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:25.120694057 +0000 UTC m=+807.316385177" watchObservedRunningTime="2026-10-30 21:42:25.121713741 +0000 UTC m=+807.317404859" Oct 30 21:42:25.882092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:25.882056 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7351-e966c7"] Oct 30 21:42:25.887016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:25.886987 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7351-e966c7"] Oct 30 21:42:27.119939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.119881 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7351-e966c7" podUID="fc63da54-3e43-43a7-8b8a-72b16ca11284" containerName="test-curl-7351-e966c7" containerID="cri-o://c418c5f0f93c77d264c696a2214a652a1e1b633a17e0856044e1f94232aeff46" gracePeriod=2 Oct 30 21:42:27.238096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.238055 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg"] Oct 30 21:42:27.239094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.239073 6808 cpu_manager.go:414] "containerMap: removing container" podUID="fc63da54-3e43-43a7-8b8a-72b16ca11284" containerName="test-curl-7351-e966c7" Oct 30 21:42:27.239171 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.239097 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc63da54-3e43-43a7-8b8a-72b16ca11284" containerName="test-curl-7351-e966c7" Oct 30 21:42:27.239318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.239305 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc63da54-3e43-43a7-8b8a-72b16ca11284" containerName="test-curl-7351-e966c7" Oct 30 21:42:27.243541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.243518 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.253733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.253711 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg"] Oct 30 21:42:27.254143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.254122 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e19227f8-openshift-default-dockercfg-x7nc6\"" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405494 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405638 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405688 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405720 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdvz\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405748 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405794 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405858 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.405929 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.407967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.406027 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510004 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510076 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510140 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510216 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510272 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510426 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510567 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510606 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.510976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.510640 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdvz\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.511921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.511378 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.511921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.511619 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.525132 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.519573 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.525132 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.520652 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.587972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.571722 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdvz\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.587972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.572805 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.587972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.573043 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.588860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.588792 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.589360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.589308 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy\") pod \"e2e-ait-e19227f8-openshift-default-57799c945-89phg\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:27.595468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:27.595259 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:28.176495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:28.175798 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg"] Oct 30 21:42:28.208894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:28.208589 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:28.208894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:28.208773 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:28.208894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:28.208868 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:42:29.193275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.193178 6808 generic.go:356] "Generic (PLEG): container finished" podID="fc63da54-3e43-43a7-8b8a-72b16ca11284" containerID="c418c5f0f93c77d264c696a2214a652a1e1b633a17e0856044e1f94232aeff46" exitCode=137 Oct 30 21:42:29.195054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.195024 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" event={"ID":"f755c347-e05c-482f-b75b-8b31857a4a37","Type":"ContainerStarted","Data":"c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4"} Oct 30 21:42:29.195177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.195074 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" event={"ID":"f755c347-e05c-482f-b75b-8b31857a4a37","Type":"ContainerStarted","Data":"e307933c3e891622e3603fc7a424426519de7bf084051f6b04e812c2eddc630b"} Oct 30 21:42:29.224678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.224631 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" podStartSLOduration=2.224616814 podStartE2EDuration="2.224616814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:29.221021541 +0000 UTC m=+811.416712661" watchObservedRunningTime="2026-10-30 21:42:29.224616814 +0000 UTC m=+811.420307933" Oct 30 21:42:29.317024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.316991 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:29.386960 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.386917 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm\" (UniqueName: \"kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm\") pod \"fc63da54-3e43-43a7-8b8a-72b16ca11284\" (UID: \"fc63da54-3e43-43a7-8b8a-72b16ca11284\") " Oct 30 21:42:29.398072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.398015 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm" pod "fc63da54-3e43-43a7-8b8a-72b16ca11284" (UID: "fc63da54-3e43-43a7-8b8a-72b16ca11284"). InnerVolumeSpecName "kube-api-access-lm8dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:29.495387 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.495317 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8dm\" (UniqueName: \"kubernetes.io/projected/fc63da54-3e43-43a7-8b8a-72b16ca11284-kube-api-access-lm8dm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:29.600357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.600323 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:29.605829 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.605803 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:29.929292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:29.929264 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc63da54-3e43-43a7-8b8a-72b16ca11284" path="/var/lib/kubelet/pods/fc63da54-3e43-43a7-8b8a-72b16ca11284/volumes" Oct 30 21:42:30.225126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:30.225043 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-e966c7" Oct 30 21:42:30.226361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:30.226140 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:30.226504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:30.225743 6808 scope.go:122] "RemoveContainer" containerID="c418c5f0f93c77d264c696a2214a652a1e1b633a17e0856044e1f94232aeff46" Oct 30 21:42:30.237085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:30.230357 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:42:34.997007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:34.996974 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq"] Oct 30 21:42:35.005603 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.005577 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.014021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.013996 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq"] Oct 30 21:42:35.096126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.096101 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-938e67d1\"" Oct 30 21:42:35.096275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.096131 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-938e67d1-dockercfg-stx79\"" Oct 30 21:42:35.104928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.104903 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstdt\" (UniqueName: \"kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt\") pod \"payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.105345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.105318 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.231970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.231609 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstdt\" (UniqueName: \"kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt\") pod \"payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.231970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.231865 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.235969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.232899 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.260014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.257667 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstdt\" (UniqueName: \"kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt\") pod \"payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.355125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:35.355095 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:35.836911 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:42:35.836871 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:42:36.032969 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:42:36.032511 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05c2be1_1920_4c5e_a68c_6c747ba68d69.slice/crio-407c3665773e872716d2c87621fad3712506e1db520cd8bb80dd8c78ab50e54b WatchSource:0}: Error finding container 407c3665773e872716d2c87621fad3712506e1db520cd8bb80dd8c78ab50e54b: Status 404 returned error can't find the container with id 407c3665773e872716d2c87621fad3712506e1db520cd8bb80dd8c78ab50e54b Oct 30 21:42:36.357983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:36.357482 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq"] Oct 30 21:42:36.735969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:36.734986 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" event={"ID":"c05c2be1-1920-4c5e-a68c-6c747ba68d69","Type":"ContainerStarted","Data":"407c3665773e872716d2c87621fad3712506e1db520cd8bb80dd8c78ab50e54b"} Oct 30 21:42:36.870180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:36.869066 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:42:37.061554 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.050062 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.062226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.062187 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.147669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.147622 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.150097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.149687 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.382034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.381536 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.707022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.700037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:42:37.758069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.757353 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.758069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.757675 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.795034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.794817 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" event={"ID":"c05c2be1-1920-4c5e-a68c-6c747ba68d69","Type":"ContainerStarted","Data":"eb84019bdff36b566d55a8d76f1810846c451d1029048398cd98df30b0413615"} Oct 30 21:42:37.814089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.814051 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:37.834675 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.834632 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.856178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.856118 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" podStartSLOduration=3.856099188 podStartE2EDuration="3.856099188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:37.8550477 +0000 UTC m=+820.050738820" watchObservedRunningTime="2026-10-30 21:42:37.856099188 +0000 UTC m=+820.051790307" Oct 30 21:42:37.884366 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.884325 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:42:37.885784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:37.884812 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:42:38.063261 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.063151 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:42:38.089229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.073078 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.177292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.133218 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.230979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.179037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:38.230979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.179274 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:38.368030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.360452 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:38.368030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.360521 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:38.368030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.360560 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:42:38.368030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.360605 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:42:38.385979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.385109 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.385979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.385659 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:42:38.722980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.721839 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.722980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.722180 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:42:38.758104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.758059 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.775973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.758719 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:42:38.894974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.850144 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.922054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.905527 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:38.964976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:38.914521 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.382077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.348503 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.384349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.364324 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.443740 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.364390 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:39.443740 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.364415 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:39.443740 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.364447 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:39.444247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.444224 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:42:39.465973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.385388 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.737033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.735150 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.771983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.770309 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.948979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.922500 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.981973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:39.980843 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.419101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.419050 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:40.435978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.431027 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.479132 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.479083 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:40.479826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.479791 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:40.487981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.480313 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:40.488297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.488266 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:40.740331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.740230 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.773631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.773590 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:40.923741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.923694 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.981998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:40.981023 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.023339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.023025 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.431794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.431742 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.484532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.484485 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:41.485040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.485011 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:41.486680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.486653 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.498231 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.497774 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:41.539368 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.539107 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.540489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.540450 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.744311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.744184 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:42:41.778023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.777044 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.855076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.855029 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.941996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.935041 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.983620 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:41.983049 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.287614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.287581 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:42.314702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.314674 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:42:42.315897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.315862 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.391001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.390897 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:42:42.457140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.455916 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.500771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.471728 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:42.500771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.491896 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.500771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.491927 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:42.500771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.500702 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:42.640748 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.639495 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:42:42.678662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.678635 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:42:42.752235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.752206 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:42.770972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.770924 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:42:42.984268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:42.984168 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.477982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:43.476590 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.495182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:43.495102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.502105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:43.502068 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:43.988075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:43.987056 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.314031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.313079 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.508772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.483366 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.508772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.487071 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:44.508772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.490030 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.508772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.497167 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:44.508772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.497718 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:44.605075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.571536 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:42:44.605075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.574045 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:44.739978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.732514 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:44.743972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.740247 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.749752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.740436 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:44.803970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.798260 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.803970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.798589 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 21:42:44.803970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.799591 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 21:42:44.803970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.799896 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:42:45.071030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:44.934823 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.090980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.076409 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.325024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.319079 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.333973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.331708 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:45.484089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.484046 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:45.553765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.495737 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.553765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.499590 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.553765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.515239 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:45.553765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.541317 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:42:45.553765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.549455 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:42:45.553765 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.549543 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:45.783970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.783370 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:45.783970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.783868 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:45.801252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.801212 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.835971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.784136 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:42:46.024977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:45.989303 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.184033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.107389 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.420991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.409081 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcv85" podUID="d4c20eec-664a-4952-9368-7f8e414ce138" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.28:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.442653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.441213 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 21:42:46.501165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.493013 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:46.501165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.493092 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.513973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.513275 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.577343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.568067 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.583924 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.583878 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:46.584532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.584498 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.619082 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.484883 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.752583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.749316 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:42:46.752583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.749377 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:42:46.794812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.794770 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.795841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.795811 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:46.821972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.809872 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.890978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:46.889252 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 21:42:47.016383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.016286 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:47.183289 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.183246 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:47.604973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.587931 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:47.604973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.588367 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:47.604973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.588629 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:47.604973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.594250 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:42:47.604973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.594616 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:47.727974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.701507 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:47.807356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.801044 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:42:47.807356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.801325 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:42:47.847193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.847144 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:47.870026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:47.849473 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 21:42:48.192236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.178386 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:48.204686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.204652 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:48.271641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.271613 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:42:48.271903 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.271889 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:42:48.272045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.272035 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:42:48.272149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.272141 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:42:48.278377 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.278351 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:42:48.279576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.279554 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:42:48.288092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.288069 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:42:48.331012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.330944 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:42:48.333285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.333260 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:42:48.340576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:48.340553 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:42:49.206980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.205422 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:42:49.800160 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.800125 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc"] Oct 30 21:42:49.805615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.805591 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:49.812592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.812553 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-938e67d1-serving-cert\"" Oct 30 21:42:49.818242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.818215 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc"] Oct 30 21:42:49.950248 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.950206 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m88l\" (UniqueName: \"kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:49.950824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.950792 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:49.950986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.950968 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:49.951121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:49.951104 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.056241 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.056146 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m88l\" (UniqueName: \"kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.076979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.056739 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.077264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.077237 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.077395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.077376 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.078479 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.078452 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.078632 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:42:50.073117 6808 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-serving-cert: secret "maas-api-e2e-migrate-938e67d1-serving-cert" not found Oct 30 21:42:50.078807 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:42:50.078795 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls podName:d9a6a60b-32e6-4363-a2b7-a86a0a91e223 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:50.578773149 +0000 UTC m=+832.774464248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls") pod "maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" (UID: "d9a6a60b-32e6-4363-a2b7-a86a0a91e223") : secret "maas-api-e2e-migrate-938e67d1-serving-cert" not found Oct 30 21:42:50.088544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.088517 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.097685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.097248 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m88l\" (UniqueName: \"kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.628971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.606380 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.630138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.627055 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\") pod \"maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:50.725014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:50.724982 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:51.100349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:51.100322 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc"] Oct 30 21:42:51.101788 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:42:51.101760 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a6a60b_32e6_4363_a2b7_a86a0a91e223.slice/crio-9bb0b05becd4756a1ac3ef63f1975785fac9da06b93b3529ecdb9ab91b6e67d5 WatchSource:0}: Error finding container 9bb0b05becd4756a1ac3ef63f1975785fac9da06b93b3529ecdb9ab91b6e67d5: Status 404 returned error can't find the container with id 9bb0b05becd4756a1ac3ef63f1975785fac9da06b93b3529ecdb9ab91b6e67d5 Oct 30 21:42:51.298153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:51.298063 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" event={"ID":"d9a6a60b-32e6-4363-a2b7-a86a0a91e223","Type":"ContainerStarted","Data":"9bb0b05becd4756a1ac3ef63f1975785fac9da06b93b3529ecdb9ab91b6e67d5"} Oct 30 21:42:52.308373 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:52.308338 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" event={"ID":"d9a6a60b-32e6-4363-a2b7-a86a0a91e223","Type":"ContainerStarted","Data":"1cb0a5493d7fffdf93895d807ac257adba012b51de01415cb528e8a5fbcc216d"} Oct 30 21:42:52.308827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:52.308467 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:42:52.333099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:52.333040 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" podStartSLOduration=2.9075308079999997 podStartE2EDuration="3.333022878s" totalImagesPullingTime="425.49207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:52.329362383 +0000 UTC m=+834.525053502" watchObservedRunningTime="2026-10-30 21:42:52.333022878 +0000 UTC m=+834.528713998" Oct 30 21:42:58.364778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:42:58.364749 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:43:02.576689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:02.576641 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:02.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:02.610123 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:02.663411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:02.616563 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:03.229105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.228051 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.249034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.248038 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:03.439977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.434041 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.495364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.495258 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:03.495660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.495610 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.582119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.582060 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.583931 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.583892 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:03.584706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.584688 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:03.709349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.709298 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:03.729973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.713393 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.871406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.732839 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:43:03.871677 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.871654 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:43:03.872928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.817942 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.987335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.863862 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:03.988147 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:03.988110 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.232289 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.232247 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:04.249982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.246093 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:43:04.255803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.251434 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.286050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.277281 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:43:04.474976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.439034 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.475670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.475645 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:43:04.616113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.597067 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.616113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.597563 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:43:04.616113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.598202 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:04.659731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.657543 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 21:43:04.718157 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.718112 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.767137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.767091 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 21:43:04.882031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.879747 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:04.909974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.906998 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.992776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.992071 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.994035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.993456 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:43:05.016243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.994638 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:43:05.016243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.994700 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:43:05.016243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:04.996853 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.017085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.017059 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:43:05.270033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.257037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:05.270033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.266005 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:05.550507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.550394 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.874657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.779875 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.892612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.779919 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.922500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.916059 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.922500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.916741 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:06.010300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.992982 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.010300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.993836 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.010300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:05.999347 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.241443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.241234 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:06.242714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.241730 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.259741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.259690 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.307370 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.307326 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:06.488085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.488034 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.738209 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:43:06.738166 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:43:06.787399 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.787341 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.879220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.879172 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.915020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.913420 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.947996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.922025 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.947996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.922941 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.975002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.972477 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.001029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.997322 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.001029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:06.997706 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:07.002354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.002328 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.261187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.261074 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.335968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.321063 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.335968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.330403 6808 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:07.335968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.330457 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.496375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.496345 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:07.851972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.808085 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.851972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.808477 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 21:43:07.871097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.871036 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs"] Oct 30 21:43:07.966986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.953053 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:07.998973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.994024 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.998973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.997163 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:07.998973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.998088 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.998973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:07.998352 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.031030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.004041 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.102555 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.102466 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs"] Oct 30 21:43:08.107419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.107379 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.251989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.218211 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a5af9561-openshift-default-dockercfg-92f8p\"" Oct 30 21:43:08.252260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.252226 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrkm\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.252550 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.252524 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.252742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.252724 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.252941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.252924 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.257113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.257085 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.341975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.309414 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.342322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.342291 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.392192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.392166 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.392540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.392518 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.393121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.328166 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.393738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.294535 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:08.550654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.531495 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.555035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.554998 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.555342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.555323 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.555503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.555486 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.580105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.560403 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.580991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.560639 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.581173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.580566 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrkm\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.581454 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.581416 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.581682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.581663 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.581893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.581876 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.582101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.582085 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.612981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.582470 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.613226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.582682 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.613404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.584159 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.613516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.585294 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.613605 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.612573 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrkm\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.621992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.614061 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.648768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.648203 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:08.661985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.654084 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token\") pod \"e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.775350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.775315 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:08.809551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.809031 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.969768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.967622 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.995555 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.995510 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.000640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.999231 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.000640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.999550 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:09.000640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:08.999787 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:09.004964 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:09.004920 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.856000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:09.855931 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:10.700521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:09.856268 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.743656 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.018273 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.025131 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.118499 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.118651 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.123495 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.125048 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.125148 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.125322 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:10.799016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.514002 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:10.889376 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.889309 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:10.889784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:10.889752 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:11.126543 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:43:11.126421 6808 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.101s" Oct 30 21:43:11.126797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.126769 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs"] Oct 30 21:43:11.127021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.127005 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:11.188482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.188225 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.411249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.399146 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq"] Oct 30 21:43:11.682168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.682063 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.699151 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.699110 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.707973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.704568 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.863311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.846712 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:43:11.863311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.856321 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:11.863311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.857424 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.863311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.857727 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:11.863311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.858073 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:11.973000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.926027 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:11.973000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.926349 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:11.973000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:11.930315 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:12.152036 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.148285 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.322915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.322663 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.324914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.324873 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" event={"ID":"e454edad-5cc3-4ad5-99b5-fde52146d6ed","Type":"ContainerStarted","Data":"2df0207be0bb195b612c50c88f8f2a08e7f4bbab8c41e627a4eb80d2e597bf6e"} Oct 30 21:43:12.325233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.325207 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:12.772571 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.756165 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:43:12.820061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.798046 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:43:12.864147 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.864102 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.865410 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.865377 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:12.881484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.793024 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.882420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.870013 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.931393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.931334 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.932252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:12.932215 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:13.465988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.460511 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:43:13.480563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.480531 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:43:13.518289 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.517818 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:43:13.625009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.621570 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" event={"ID":"e454edad-5cc3-4ad5-99b5-fde52146d6ed","Type":"ContainerStarted","Data":"7b2e4271cae2faa377277d7d7770c19a7ea09fe62b18d66a6579f6ef0ccd8207"} Oct 30 21:43:13.780268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.779810 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:13.831999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.831972 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:43:13.836004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.835982 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:43:13.857986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:13.848314 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:14.012100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.012052 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:14.230503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.230408 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" podStartSLOduration=7.230383508 podStartE2EDuration="7.230383508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:14.121320027 +0000 UTC m=+856.317011179" watchObservedRunningTime="2026-10-30 21:43:14.230383508 +0000 UTC m=+856.426074629" Oct 30 21:43:14.279147 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.279109 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:43:14.640671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.640643 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:14.642541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.642524 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:43:14.757774 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.757679 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc"] Oct 30 21:43:14.758406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:14.758375 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" podUID="d9a6a60b-32e6-4363-a2b7-a86a0a91e223" containerName="maas-api" containerID="cri-o://1cb0a5493d7fffdf93895d807ac257adba012b51de01415cb528e8a5fbcc216d" gracePeriod=30 Oct 30 21:43:15.943313 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:15.912866 6808 generic.go:356] "Generic (PLEG): container finished" podID="d9a6a60b-32e6-4363-a2b7-a86a0a91e223" containerID="1cb0a5493d7fffdf93895d807ac257adba012b51de01415cb528e8a5fbcc216d" exitCode=0 Oct 30 21:43:15.943313 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:15.916084 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" event={"ID":"d9a6a60b-32e6-4363-a2b7-a86a0a91e223","Type":"ContainerDied","Data":"1cb0a5493d7fffdf93895d807ac257adba012b51de01415cb528e8a5fbcc216d"} Oct 30 21:43:16.075202 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.075181 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:43:16.104249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.104215 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca\") pod \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " Oct 30 21:43:16.104458 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.104409 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\") pod \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " Oct 30 21:43:16.104545 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.104475 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l\" (UniqueName: \"kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l\") pod \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " Oct 30 21:43:16.104607 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.104596 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs\") pod \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\" (UID: \"d9a6a60b-32e6-4363-a2b7-a86a0a91e223\") " Oct 30 21:43:16.104665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.104604 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca" pod "d9a6a60b-32e6-4363-a2b7-a86a0a91e223" (UID: "d9a6a60b-32e6-4363-a2b7-a86a0a91e223"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:16.105380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.105354 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:16.107167 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.107145 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls" pod "d9a6a60b-32e6-4363-a2b7-a86a0a91e223" (UID: "d9a6a60b-32e6-4363-a2b7-a86a0a91e223"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:16.107992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.107933 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l" pod "d9a6a60b-32e6-4363-a2b7-a86a0a91e223" (UID: "d9a6a60b-32e6-4363-a2b7-a86a0a91e223"). InnerVolumeSpecName "kube-api-access-9m88l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:16.108711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.108559 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs" pod "d9a6a60b-32e6-4363-a2b7-a86a0a91e223" (UID: "d9a6a60b-32e6-4363-a2b7-a86a0a91e223"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:16.207767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.207001 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:16.207767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.207029 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m88l\" (UniqueName: \"kubernetes.io/projected/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-kube-api-access-9m88l\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:16.207767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.207039 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a6a60b-32e6-4363-a2b7-a86a0a91e223-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:16.491206 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.491076 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" podUID="c05c2be1-1920-4c5e-a68c-6c747ba68d69" containerName="payload-pre-processing" containerID="cri-o://eb84019bdff36b566d55a8d76f1810846c451d1029048398cd98df30b0413615" gracePeriod=25 Oct 30 21:43:16.579684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.579646 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67"] Oct 30 21:43:16.581127 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.581105 6808 cpu_manager.go:414] "containerMap: removing container" podUID="d9a6a60b-32e6-4363-a2b7-a86a0a91e223" containerName="maas-api" Oct 30 21:43:16.581305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.581290 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a6a60b-32e6-4363-a2b7-a86a0a91e223" containerName="maas-api" Oct 30 21:43:16.581592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.581579 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a6a60b-32e6-4363-a2b7-a86a0a91e223" containerName="maas-api" Oct 30 21:43:16.597272 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.597253 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.609639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.609189 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6d718596-openshift-default-dockercfg-lmzmb\"" Oct 30 21:43:16.610223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.610197 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67"] Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.612896 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.612935 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.612989 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.613048 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.613089 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.613152 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.613267 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rqh\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.613359 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.614092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.613447 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.731337 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.731278 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rqh\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.763205 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.763104 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.763741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.763716 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.764159 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.764137 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.764341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.764323 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.764476 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.764459 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.764666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.764651 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.764815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.764799 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.765098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.765067 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.793558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.766418 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.793558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.766675 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.793558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.769449 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.793558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.774189 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.793558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.792470 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.836227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.836159 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.844970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.840384 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.856566 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.856505 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.859974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.858574 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rqh\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh\") pod \"e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:16.948985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:16.943352 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:17.049767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.049353 6808 generic.go:356] "Generic (PLEG): container finished" podID="c05c2be1-1920-4c5e-a68c-6c747ba68d69" containerID="eb84019bdff36b566d55a8d76f1810846c451d1029048398cd98df30b0413615" exitCode=0 Oct 30 21:43:17.049767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.049617 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" event={"ID":"c05c2be1-1920-4c5e-a68c-6c747ba68d69","Type":"ContainerDied","Data":"eb84019bdff36b566d55a8d76f1810846c451d1029048398cd98df30b0413615"} Oct 30 21:43:17.054984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.053994 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" Oct 30 21:43:17.069418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.069379 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc" event={"ID":"d9a6a60b-32e6-4363-a2b7-a86a0a91e223","Type":"ContainerDied","Data":"9bb0b05becd4756a1ac3ef63f1975785fac9da06b93b3529ecdb9ab91b6e67d5"} Oct 30 21:43:17.069726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.069699 6808 scope.go:122] "RemoveContainer" containerID="1cb0a5493d7fffdf93895d807ac257adba012b51de01415cb528e8a5fbcc216d" Oct 30 21:43:17.479880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.479843 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8"] Oct 30 21:43:17.744342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.744299 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:17.766356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.766324 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:17.825014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.815250 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a5af9561-dockercfg-w2m8z\"" Oct 30 21:43:17.858218 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.858190 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8"] Oct 30 21:43:17.906223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.895385 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj"] Oct 30 21:43:17.906223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.906082 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:17.944970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.907765 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a5af9561\"" Oct 30 21:43:17.944970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.924838 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzpz\" (UniqueName: \"kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz\") pod \"payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:17.944970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:17.925032 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:18.251886 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.250926 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:18.288330 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.261586 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:18.288330 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.272576 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:18.288330 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.276683 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.395337 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.395304 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:18.395574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.395557 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume\") pod \"payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.395868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.395841 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzbj\" (UniqueName: \"kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj\") pod \"payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.396214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.396193 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzpz\" (UniqueName: \"kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz\") pod \"payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:18.397488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.397453 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:18.404214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.404194 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc"] Oct 30 21:43:18.435118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.412068 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzpz\" (UniqueName: \"kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz\") pod \"payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:18.435118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.427002 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-938e67d1-bfb75d76f-lqvnc"] Oct 30 21:43:18.435118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.431807 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:43:18.439943 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.439881 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj"] Oct 30 21:43:18.454038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.453340 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" event={"ID":"c05c2be1-1920-4c5e-a68c-6c747ba68d69","Type":"ContainerDied","Data":"407c3665773e872716d2c87621fad3712506e1db520cd8bb80dd8c78ab50e54b"} Oct 30 21:43:18.454038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.453398 6808 scope.go:122] "RemoveContainer" containerID="eb84019bdff36b566d55a8d76f1810846c451d1029048398cd98df30b0413615" Oct 30 21:43:18.454038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.453511 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq" Oct 30 21:43:18.499052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.496899 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume\") pod \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " Oct 30 21:43:18.499052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.497123 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt\" (UniqueName: \"kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt\") pod \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\" (UID: \"c05c2be1-1920-4c5e-a68c-6c747ba68d69\") " Oct 30 21:43:18.499052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.497757 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume\") pod \"payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.499052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.498187 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzbj\" (UniqueName: \"kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj\") pod \"payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.499052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.498935 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume" pod "c05c2be1-1920-4c5e-a68c-6c747ba68d69" (UID: "c05c2be1-1920-4c5e-a68c-6c747ba68d69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:18.503075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.501896 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume\") pod \"payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.504033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.503993 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:18.515093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.515067 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt" pod "c05c2be1-1920-4c5e-a68c-6c747ba68d69" (UID: "c05c2be1-1920-4c5e-a68c-6c747ba68d69"). InnerVolumeSpecName "kube-api-access-cstdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:18.518218 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.518175 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzbj\" (UniqueName: \"kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj\") pod \"payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.569609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.569089 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:18.569609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.569177 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:18.569609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.569222 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:18.593880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.593838 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67"] Oct 30 21:43:18.600645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.600599 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstdt\" (UniqueName: \"kubernetes.io/projected/c05c2be1-1920-4c5e-a68c-6c747ba68d69-kube-api-access-cstdt\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:18.600645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.600624 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05c2be1-1920-4c5e-a68c-6c747ba68d69-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:18.724847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.724541 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:18.869899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.869837 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq"] Oct 30 21:43:18.876144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:18.875734 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-938e67d1-65f4d6898c-wmjrq"] Oct 30 21:43:19.081381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.081306 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8"] Oct 30 21:43:19.283546 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.283512 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj"] Oct 30 21:43:19.505929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.503891 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" event={"ID":"4e8975bd-1599-488d-8d15-afd27382d869","Type":"ContainerStarted","Data":"b2f2303a2f7c166f5726c338d0780b2139485d6b9dc9649be6bd610902353c98"} Oct 30 21:43:19.505929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.503977 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" event={"ID":"4e8975bd-1599-488d-8d15-afd27382d869","Type":"ContainerStarted","Data":"6b45f1930935cc86e6396f65eaf35ae3ca64e2953d4324acba10ea8a549da598"} Oct 30 21:43:19.505929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.505228 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:19.510531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.508314 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" event={"ID":"741950cd-5d73-4e5f-b730-b55fdd2ab002","Type":"ContainerStarted","Data":"de094c611d09485f9c62f47432011a59b1cd8717158ec48bb29806c0ce34a8e6"} Oct 30 21:43:19.510531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.508366 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" event={"ID":"741950cd-5d73-4e5f-b730-b55fdd2ab002","Type":"ContainerStarted","Data":"c044e2e2324fbb3ec0035ef81fb3efddaf91b28a3aa2fb034df5cf3e7060132d"} Oct 30 21:43:19.510531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.509295 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:19.514669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.513497 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" event={"ID":"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5","Type":"ContainerStarted","Data":"ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843"} Oct 30 21:43:19.514669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.513545 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" event={"ID":"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5","Type":"ContainerStarted","Data":"bf48e31474561ada013a9c344be45323478f87ddc09054d8f36d3c376165a862"} Oct 30 21:43:19.555961 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.555847 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" podStartSLOduration=2.555820308 podStartE2EDuration="2.555820308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:19.524942562 +0000 UTC m=+861.720633702" watchObservedRunningTime="2026-10-30 21:43:19.555820308 +0000 UTC m=+861.751511480" Oct 30 21:43:19.606648 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.606579 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" podStartSLOduration=3.60654965 podStartE2EDuration="3.60654965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:19.590075186 +0000 UTC m=+861.785766306" watchObservedRunningTime="2026-10-30 21:43:19.60654965 +0000 UTC m=+861.802240770" Oct 30 21:43:19.624225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.624028 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" podStartSLOduration=2.624008525 podStartE2EDuration="2.624008525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:19.62140678 +0000 UTC m=+861.817097911" watchObservedRunningTime="2026-10-30 21:43:19.624008525 +0000 UTC m=+861.819699646" Oct 30 21:43:19.982629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.981445 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05c2be1-1920-4c5e-a68c-6c747ba68d69" path="/var/lib/kubelet/pods/c05c2be1-1920-4c5e-a68c-6c747ba68d69/volumes" Oct 30 21:43:19.982629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.982173 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a6a60b-32e6-4363-a2b7-a86a0a91e223" path="/var/lib/kubelet/pods/d9a6a60b-32e6-4363-a2b7-a86a0a91e223/volumes" Oct 30 21:43:19.982629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.982485 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:19.982629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:19.982571 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:20.540002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:20.538749 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:20.540002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:20.539682 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:43:23.059972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.026000 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl"] Oct 30 21:43:23.126639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.112697 6808 cpu_manager.go:414] "containerMap: removing container" podUID="c05c2be1-1920-4c5e-a68c-6c747ba68d69" containerName="payload-pre-processing" Oct 30 21:43:23.226191 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.226153 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05c2be1-1920-4c5e-a68c-6c747ba68d69" containerName="payload-pre-processing" Oct 30 21:43:23.230394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.230370 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05c2be1-1920-4c5e-a68c-6c747ba68d69" containerName="payload-pre-processing" Oct 30 21:43:23.305104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.287001 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl"] Oct 30 21:43:23.305104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.287065 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z"] Oct 30 21:43:23.321673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.321579 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.331430 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.331399 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z"] Oct 30 21:43:23.331802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.331718 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.362084 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.361982 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.369356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.369320 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.370094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.370016 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m999l\" (UniqueName: \"kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l\") pod \"payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.370253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.370199 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlt4p\" (UniqueName: \"kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p\") pod \"payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.372179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.372152 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6d718596\"" Oct 30 21:43:23.372394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.372270 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6d718596-dockercfg-zgngx\"" Oct 30 21:43:23.481128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.476537 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlt4p\" (UniqueName: \"kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p\") pod \"payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.481128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.476629 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.481128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.476794 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.481128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.476992 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m999l\" (UniqueName: \"kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l\") pod \"payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.481128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.478997 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.481128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.480124 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.508712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.508540 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlt4p\" (UniqueName: \"kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p\") pod \"payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:23.551175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.549036 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m999l\" (UniqueName: \"kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l\") pod \"payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.714013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.703257 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:23.714013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:23.703736 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:24.182245 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:43:24.182197 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd52cd12_8c2e_490c_af55_fbb9596d22dc.slice/crio-1c4f37b5e59b85fd79e0d512ed35f86092a1694920d094dd239f59c6b97c95b4 WatchSource:0}: Error finding container 1c4f37b5e59b85fd79e0d512ed35f86092a1694920d094dd239f59c6b97c95b4: Status 404 returned error can't find the container with id 1c4f37b5e59b85fd79e0d512ed35f86092a1694920d094dd239f59c6b97c95b4 Oct 30 21:43:24.187234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.184488 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z"] Oct 30 21:43:24.188776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.188052 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl"] Oct 30 21:43:24.811247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.811206 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" event={"ID":"bd52cd12-8c2e-490c-af55-fbb9596d22dc","Type":"ContainerStarted","Data":"dd758a024df872e8d05be87b8a5902205a3ddfc4d14f57493d6bf1d68f89193a"} Oct 30 21:43:24.811426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.811265 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" event={"ID":"bd52cd12-8c2e-490c-af55-fbb9596d22dc","Type":"ContainerStarted","Data":"1c4f37b5e59b85fd79e0d512ed35f86092a1694920d094dd239f59c6b97c95b4"} Oct 30 21:43:24.812238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.812213 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:24.813847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.813825 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" event={"ID":"758b393e-7ff8-4e4b-a608-3e1288b3dc21","Type":"ContainerStarted","Data":"e1d5336aa74c84facfd0c6765f867ad310e2017bade5172546d6038e395f6663"} Oct 30 21:43:24.813981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.813868 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" event={"ID":"758b393e-7ff8-4e4b-a608-3e1288b3dc21","Type":"ContainerStarted","Data":"4be00750de34e97d74b52c78b4fc86b3815c0be3d55dd0c63a9567bc68a50118"} Oct 30 21:43:24.814587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.814566 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:24.850992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.850920 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" podStartSLOduration=2.850902909 podStartE2EDuration="2.850902909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:24.834362549 +0000 UTC m=+867.030053669" watchObservedRunningTime="2026-10-30 21:43:24.850902909 +0000 UTC m=+867.046594031" Oct 30 21:43:24.885755 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:24.885663 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" podStartSLOduration=2.885635205 podStartE2EDuration="2.885635205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:24.868440699 +0000 UTC m=+867.064131819" watchObservedRunningTime="2026-10-30 21:43:24.885635205 +0000 UTC m=+867.081326329" Oct 30 21:43:31.332981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.284062 6808 patch_prober.go:29] interesting pod/e2e-ait-e19227f8-openshift-default-57799c945-89phg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:31.332981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.332143 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" podUID="f755c347-e05c-482f-b75b-8b31857a4a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:31.374969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.374250 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg"] Oct 30 21:43:31.374969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.374492 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" podUID="f755c347-e05c-482f-b75b-8b31857a4a37" containerName="istio-proxy" containerID="cri-o://c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4" gracePeriod=30 Oct 30 21:43:31.532982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.521939 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:31.582966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.581333 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:31.589804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:31.585213 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:32.167393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.167356 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc"] Oct 30 21:43:32.211537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.211506 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc"] Oct 30 21:43:32.211867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.211846 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.232941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.232915 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-484f7514-openshift-default-dockercfg-mnvwq\"" Oct 30 21:43:32.323163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323132 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323277 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323444 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323429 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn66r\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323490 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323477 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323576 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323627 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323773 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323757 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323788 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.323906 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.323895 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.426899 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.426978 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427058 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427115 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427204 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427315 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn66r\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427371 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427411 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.427561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.427457 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.437989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.430347 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.437989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.431019 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.437989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.431270 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.437989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.431827 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.437989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.432387 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.451587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.451546 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.472986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.463037 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.485989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.484407 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.506985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.502230 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn66r\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r\") pod \"e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:32.576977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:32.574257 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:33.532590 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:33.532561 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc"] Oct 30 21:43:33.533446 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:43:33.533404 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2b22dd_a079_464f_8a55_8f782061df28.slice/crio-0b50b1768ee48d363efa5849897fa98ea10c6bd1ff123e091c2ebd1eab25656b WatchSource:0}: Error finding container 0b50b1768ee48d363efa5849897fa98ea10c6bd1ff123e091c2ebd1eab25656b: Status 404 returned error can't find the container with id 0b50b1768ee48d363efa5849897fa98ea10c6bd1ff123e091c2ebd1eab25656b Oct 30 21:43:33.537106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:33.537062 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:33.537219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:33.537153 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:33.537219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:33.537201 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:43:34.025413 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:34.025375 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" event={"ID":"2c2b22dd-a079-464f-8a55-8f782061df28","Type":"ContainerStarted","Data":"2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76"} Oct 30 21:43:34.025797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:34.025772 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" event={"ID":"2c2b22dd-a079-464f-8a55-8f782061df28","Type":"ContainerStarted","Data":"0b50b1768ee48d363efa5849897fa98ea10c6bd1ff123e091c2ebd1eab25656b"} Oct 30 21:43:34.577597 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:34.577527 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:34.798969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:34.798732 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:34.839037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:34.838118 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" podStartSLOduration=2.838098345 podStartE2EDuration="2.838098345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:34.052907112 +0000 UTC m=+876.248598232" watchObservedRunningTime="2026-10-30 21:43:34.838098345 +0000 UTC m=+877.033789468" Oct 30 21:43:35.018321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:35.014997 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:35.019449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:35.019187 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:43:35.706422 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:43:35.706376 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:43:36.666924 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.666903 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:43:36.677724 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.677701 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.677860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.677754 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.677860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.677808 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.677860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.677846 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.678018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.677995 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:36.678077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678034 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.678077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678061 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.678181 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678145 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.678565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678220 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:36.678565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678234 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.678565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678264 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket\") pod \"f755c347-e05c-482f-b75b-8b31857a4a37\" (UID: \"f755c347-e05c-482f-b75b-8b31857a4a37\") " Oct 30 21:43:36.678565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678468 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:36.678565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678486 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:36.678893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678661 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:36.679010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.678991 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.679079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.679014 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.679079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.679030 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.679079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.679043 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f755c347-e05c-482f-b75b-8b31857a4a37-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.679079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.679056 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.680668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.680628 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:36.680762 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.680696 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:43:36.680807 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.680768 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:36.681254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.681236 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz" pod "f755c347-e05c-482f-b75b-8b31857a4a37" (UID: "f755c347-e05c-482f-b75b-8b31857a4a37"). InnerVolumeSpecName "kube-api-access-hzdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:36.780418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.780336 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f755c347-e05c-482f-b75b-8b31857a4a37-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.780418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.780367 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.780418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.780377 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f755c347-e05c-482f-b75b-8b31857a4a37-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.780418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.780387 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdvz\" (UniqueName: \"kubernetes.io/projected/f755c347-e05c-482f-b75b-8b31857a4a37-kube-api-access-hzdvz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:36.838463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.838437 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:43:36.838622 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:36.838559 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:43:37.031693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.031597 6808 generic.go:356] "Generic (PLEG): container finished" podID="f755c347-e05c-482f-b75b-8b31857a4a37" containerID="c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4" exitCode=0 Oct 30 21:43:37.031693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.031647 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" Oct 30 21:43:37.031918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.031686 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" event={"ID":"f755c347-e05c-482f-b75b-8b31857a4a37","Type":"ContainerDied","Data":"c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4"} Oct 30 21:43:37.031918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.031744 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg" event={"ID":"f755c347-e05c-482f-b75b-8b31857a4a37","Type":"ContainerDied","Data":"e307933c3e891622e3603fc7a424426519de7bf084051f6b04e812c2eddc630b"} Oct 30 21:43:37.031918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.031775 6808 scope.go:122] "RemoveContainer" containerID="c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4" Oct 30 21:43:37.043294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.043279 6808 scope.go:122] "RemoveContainer" containerID="c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4" Oct 30 21:43:37.043559 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:43:37.043539 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4\": container with ID starting with c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4 not found: ID does not exist" containerID="c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4" Oct 30 21:43:37.043605 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.043568 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4"} err="failed to get container status \"c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4\": rpc error: code = NotFound desc = could not find container \"c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4\": container with ID starting with c8958fa49b02d22c85c558cba782528730fc7a017eb89d1a4406a352fc0545b4 not found: ID does not exist" Oct 30 21:43:37.166081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.163659 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg"] Oct 30 21:43:37.214088 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.214055 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e19227f8-openshift-default-57799c945-89phg"] Oct 30 21:43:37.978222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:37.978193 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f755c347-e05c-482f-b75b-8b31857a4a37" path="/var/lib/kubelet/pods/f755c347-e05c-482f-b75b-8b31857a4a37/volumes" Oct 30 21:43:45.764568 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:45.764529 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj"] Oct 30 21:43:45.797001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:45.796966 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8"] Oct 30 21:43:50.768162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:50.767904 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" podUID="4e8975bd-1599-488d-8d15-afd27382d869" containerName="payload-processing" containerID="cri-o://b2f2303a2f7c166f5726c338d0780b2139485d6b9dc9649be6bd610902353c98" gracePeriod=25 Oct 30 21:43:50.816094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:50.815244 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" podUID="741950cd-5d73-4e5f-b730-b55fdd2ab002" containerName="payload-pre-processing" containerID="cri-o://de094c611d09485f9c62f47432011a59b1cd8717158ec48bb29806c0ce34a8e6" gracePeriod=25 Oct 30 21:43:50.910976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:50.910916 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.023061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.018052 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.023061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.018449 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:43:51.023061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.018917 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:51.107984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.080335 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.282207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.282115 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:51.370028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.369060 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.533595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.533491 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.572962 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:43:51.572919 6808 logging.go:55] [core] [Channel #293 SubChannel #294]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.84:50052", ServerName: "10.134.0.84:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.84:50052: connect: connection refused" Oct 30 21:43:51.573791 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:43:51.573773 6808 logging.go:55] [core] [Channel #295 SubChannel #296]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.83:50052", ServerName: "10.134.0.83:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.83:50052: connect: connection refused" Oct 30 21:43:51.753998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.749441 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:51.801038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.800055 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.841974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.834717 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.892977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.891401 6808 generic.go:356] "Generic (PLEG): container finished" podID="4e8975bd-1599-488d-8d15-afd27382d869" containerID="b2f2303a2f7c166f5726c338d0780b2139485d6b9dc9649be6bd610902353c98" exitCode=0 Oct 30 21:43:51.892977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.891555 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" event={"ID":"4e8975bd-1599-488d-8d15-afd27382d869","Type":"ContainerDied","Data":"b2f2303a2f7c166f5726c338d0780b2139485d6b9dc9649be6bd610902353c98"} Oct 30 21:43:51.905976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.905463 6808 generic.go:356] "Generic (PLEG): container finished" podID="741950cd-5d73-4e5f-b730-b55fdd2ab002" containerID="de094c611d09485f9c62f47432011a59b1cd8717158ec48bb29806c0ce34a8e6" exitCode=0 Oct 30 21:43:51.905976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.905548 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" event={"ID":"741950cd-5d73-4e5f-b730-b55fdd2ab002","Type":"ContainerDied","Data":"de094c611d09485f9c62f47432011a59b1cd8717158ec48bb29806c0ce34a8e6"} Oct 30 21:43:51.920087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.920038 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.963278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:51.941923 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.020319 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.020269 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.023625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.020574 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:52.035938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.021302 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.114471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.114381 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:43:52.115190 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.115170 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:43:52.116330 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.116312 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:43:52.124651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.117521 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:43:52.316098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.300647 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.316826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.316798 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:43:52.371345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.370078 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.424230 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.398039 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:43:52.455003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.454873 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 21:43:52.608973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.577447 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" podUID="741950cd-5d73-4e5f-b730-b55fdd2ab002" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.83:50052\" within 1s: context deadline exceeded" Oct 30 21:43:52.667759 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:43:52.577535 6808 logging.go:55] [core] [Channel #295 SubChannel #296]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.83:50052", ServerName: "10.134.0.83:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.83:50052: operation was canceled" Oct 30 21:43:52.668022 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:43:52.577787 6808 logging.go:55] [core] [Channel #293 SubChannel #294]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.84:50052", ServerName: "10.134.0.84:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.84:50052: operation was canceled" Oct 30 21:43:52.696974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.577732 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" podUID="4e8975bd-1599-488d-8d15-afd27382d869" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.84:50052\" within 1s: context deadline exceeded" Oct 30 21:43:52.837973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.837463 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.837973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.837646 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:43:53.029224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.997837 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:53.029224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.998233 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:53.029224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:52.999078 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:43:53.029224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.026130 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:53.029224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.028517 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:53.122296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.091828 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:43:53.220059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.193831 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:53.238993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.238185 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:53.323104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.319024 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:53.355187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.347985 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume\") pod \"741950cd-5d73-4e5f-b730-b55fdd2ab002\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " Oct 30 21:43:53.355187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.348057 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz\" (UniqueName: \"kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz\") pod \"741950cd-5d73-4e5f-b730-b55fdd2ab002\" (UID: \"741950cd-5d73-4e5f-b730-b55fdd2ab002\") " Oct 30 21:43:53.366340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.366302 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume" pod "741950cd-5d73-4e5f-b730-b55fdd2ab002" (UID: "741950cd-5d73-4e5f-b730-b55fdd2ab002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:53.399700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.388714 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:53.500680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.500647 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741950cd-5d73-4e5f-b730-b55fdd2ab002-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:53.551399 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.551352 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz" pod "741950cd-5d73-4e5f-b730-b55fdd2ab002" (UID: "741950cd-5d73-4e5f-b730-b55fdd2ab002"). InnerVolumeSpecName "kube-api-access-wvzpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:53.620500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.620407 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzpz\" (UniqueName: \"kubernetes.io/projected/741950cd-5d73-4e5f-b730-b55fdd2ab002-kube-api-access-wvzpz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:53.913990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:53.909870 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.010784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.010691 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.011130 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.011098 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:54.011738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.011712 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:54.033101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.033063 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.034266 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.034233 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:54.154032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.152450 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.221973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.221264 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.225995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.222607 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:54.225995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.222972 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume\") pod \"4e8975bd-1599-488d-8d15-afd27382d869\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " Oct 30 21:43:54.225995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.223057 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj\" (UniqueName: \"kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj\") pod \"4e8975bd-1599-488d-8d15-afd27382d869\" (UID: \"4e8975bd-1599-488d-8d15-afd27382d869\") " Oct 30 21:43:54.315040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.275866 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume" pod "4e8975bd-1599-488d-8d15-afd27382d869" (UID: "4e8975bd-1599-488d-8d15-afd27382d869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:54.327918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.327878 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8975bd-1599-488d-8d15-afd27382d869-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:54.394033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.387080 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:54.520717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.520612 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.717151 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.717103 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 21:43:54.862972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.855890 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 21:43:54.895902 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.873268 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.895902 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.889150 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj" pod "4e8975bd-1599-488d-8d15-afd27382d869" (UID: "4e8975bd-1599-488d-8d15-afd27382d869"). InnerVolumeSpecName "kube-api-access-8dzbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:54.945047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.920217 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.945047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:54.930509 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.106969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.083992 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.123062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.107268 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:55.183076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.123458 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:55.183076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.144527 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.223200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.223144 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.336672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.258085 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:43:55.395051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.388630 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzbj\" (UniqueName: \"kubernetes.io/projected/4e8975bd-1599-488d-8d15-afd27382d869-kube-api-access-8dzbj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:55.406711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.406667 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" event={"ID":"741950cd-5d73-4e5f-b730-b55fdd2ab002","Type":"ContainerDied","Data":"c044e2e2324fbb3ec0035ef81fb3efddaf91b28a3aa2fb034df5cf3e7060132d"} Oct 30 21:43:55.407031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.407008 6808 scope.go:122] "RemoveContainer" containerID="de094c611d09485f9c62f47432011a59b1cd8717158ec48bb29806c0ce34a8e6" Oct 30 21:43:55.407417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.407397 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8" Oct 30 21:43:55.458974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.442193 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:55.676704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.625735 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:55.880163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.880101 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.884261 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.884235 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:43:56.283193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.912311 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" Oct 30 21:43:56.284024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:55.956709 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.387521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.279524 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.418050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.289145 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj" event={"ID":"4e8975bd-1599-488d-8d15-afd27382d869","Type":"ContainerDied","Data":"6b45f1930935cc86e6396f65eaf35ae3ca64e2953d4324acba10ea8a549da598"} Oct 30 21:43:56.419688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.419636 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 21:43:56.419899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.291519 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:56.420327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.291539 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:56.420730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.291565 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:56.421152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.291582 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:56.450123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.291464 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:56.450123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.446101 6808 scope.go:122] "RemoveContainer" containerID="b2f2303a2f7c166f5726c338d0780b2139485d6b9dc9649be6bd610902353c98" Oct 30 21:43:56.511094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.511050 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.536120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.535105 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8"] Oct 30 21:43:56.536120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.535146 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a5af9561-8794cccd7-68wt8"] Oct 30 21:43:56.619482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.619431 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj"] Oct 30 21:43:56.633560 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.627331 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:56.670031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.644383 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a5af9561-6cd7d7cb95-58shj"] Oct 30 21:43:56.699533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.699507 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:43:56.916971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:56.907625 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:57.068024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.054298 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:57.355641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.355535 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs"] Oct 30 21:43:57.437238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.437176 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:57.509408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.478127 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:43:57.509408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.480367 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:57.509408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.480638 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:57.509408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.480848 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:57.509408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.481074 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:43:57.581598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.580913 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" podUID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" containerName="istio-proxy" containerID="cri-o://7b2e4271cae2faa377277d7d7770c19a7ea09fe62b18d66a6579f6ef0ccd8207" gracePeriod=30 Oct 30 21:43:57.646207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.638212 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:57.767969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.759798 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z"] Oct 30 21:43:57.796446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.795694 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl"] Oct 30 21:43:57.823001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.822943 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:57.918210 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:57.913171 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:58.098132 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:43:58.085078 6808 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741950cd_5d73_4e5f_b730_b55fdd2ab002.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8975bd_1599_488d_8d15_afd27382d869.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741950cd_5d73_4e5f_b730_b55fdd2ab002.slice/crio-c044e2e2324fbb3ec0035ef81fb3efddaf91b28a3aa2fb034df5cf3e7060132d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8975bd_1599_488d_8d15_afd27382d869.slice/crio-6b45f1930935cc86e6396f65eaf35ae3ca64e2953d4324acba10ea8a549da598\": RecentStats: unable to find data in memory cache]" Oct 30 21:43:58.587975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.584765 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:58.755800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.755752 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:58.812969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.799923 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:43:58.845620 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.845129 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:43:58.846158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.800530 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:43:58.846746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.800748 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:43:58.847122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.800965 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:58.889191 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.864871 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:58.980973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:58.800297 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:59.052972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.052430 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:59.482146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.474919 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8975bd-1599-488d-8d15-afd27382d869" path="/var/lib/kubelet/pods/4e8975bd-1599-488d-8d15-afd27382d869/volumes" Oct 30 21:43:59.482146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.475826 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741950cd-5d73-4e5f-b730-b55fdd2ab002" path="/var/lib/kubelet/pods/741950cd-5d73-4e5f-b730-b55fdd2ab002/volumes" Oct 30 21:43:59.482146 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:43:59.481298 6808 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.482s" Oct 30 21:43:59.482146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.481626 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:43:59.618872 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.614104 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:43:59.618872 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.614502 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:43:59.770022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.765594 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:59.825595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.822902 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.889136 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2"] Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890218 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f755c347-e05c-482f-b75b-8b31857a4a37" containerName="istio-proxy" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890243 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f755c347-e05c-482f-b75b-8b31857a4a37" containerName="istio-proxy" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890266 6808 cpu_manager.go:414] "containerMap: removing container" podUID="4e8975bd-1599-488d-8d15-afd27382d869" containerName="payload-processing" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890274 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8975bd-1599-488d-8d15-afd27382d869" containerName="payload-processing" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890300 6808 cpu_manager.go:414] "containerMap: removing container" podUID="741950cd-5d73-4e5f-b730-b55fdd2ab002" containerName="payload-pre-processing" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890307 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741950cd-5d73-4e5f-b730-b55fdd2ab002" containerName="payload-pre-processing" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890446 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f755c347-e05c-482f-b75b-8b31857a4a37" containerName="istio-proxy" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890459 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8975bd-1599-488d-8d15-afd27382d869" containerName="payload-processing" Oct 30 21:43:59.930079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:43:59.890472 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="741950cd-5d73-4e5f-b730-b55fdd2ab002" containerName="payload-pre-processing" Oct 30 21:44:00.033601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.033510 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2"] Oct 30 21:44:00.292601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.292505 6808 patch_prober.go:29] interesting pod/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:44:00.292850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.292816 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" podUID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:44:00.293018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.039427 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.039446 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.103916 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.103980 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.111150 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.200179 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.039405 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.369108 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470039 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470171 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470207 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470288 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470388 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470586 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470649 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs2q\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.470705 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.478629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.039371 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:44:00.492974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.489844 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2b36c1-openshift-default-dockercfg-kpffw\"" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612354 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612420 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612513 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612545 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612610 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612694 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612860 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612920 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs2q\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.612993 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.613811 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.614110 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.614747 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.631994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.625928 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:00.642993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.635599 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.646477 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.646453 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.696973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.678738 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.712018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.711987 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.720002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.719973 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.724229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.724205 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:44:00.726542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.726522 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:44:00.727919 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.727893 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:44:00.728213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.728196 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:44:00.733733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.733714 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:44:00.738696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.738624 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs2q\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q\") pod \"e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:00.771888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.771849 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:00.828102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.825023 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:00.828102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.826828 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:44:00.914298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.914268 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:44:00.950122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.950092 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:44:00.950386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.950371 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:44:00.984506 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:00.984459 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:44:01.028574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.028485 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:01.061982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.054408 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:01.075721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.075694 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:44:01.120332 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.119901 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:44:01.506718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.506689 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2"] Oct 30 21:44:01.510397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.510357 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:01.510487 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.510442 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:01.510531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.510481 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:01.822009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:01.812976 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" event={"ID":"219686ae-b68b-4d6d-b32a-0de8543b2064","Type":"ContainerStarted","Data":"3ebf314287d73c6deb9d4c88fe4f0c6cd0b155ef4e8ff92ee8f0e104c0a5b4f3"} Oct 30 21:44:02.777973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:02.777192 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" podUID="bd52cd12-8c2e-490c-af55-fbb9596d22dc" containerName="payload-processing" containerID="cri-o://dd758a024df872e8d05be87b8a5902205a3ddfc4d14f57493d6bf1d68f89193a" gracePeriod=25 Oct 30 21:44:02.798396 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:02.797762 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" podUID="758b393e-7ff8-4e4b-a608-3e1288b3dc21" containerName="payload-pre-processing" containerID="cri-o://e1d5336aa74c84facfd0c6765f867ad310e2017bade5172546d6038e395f6663" gracePeriod=25 Oct 30 21:44:02.828863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:02.828826 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" event={"ID":"219686ae-b68b-4d6d-b32a-0de8543b2064","Type":"ContainerStarted","Data":"97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b"} Oct 30 21:44:02.832889 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:02.832858 6808 generic.go:356] "Generic (PLEG): container finished" podID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" containerID="7b2e4271cae2faa377277d7d7770c19a7ea09fe62b18d66a6579f6ef0ccd8207" exitCode=0 Oct 30 21:44:02.833063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:02.832983 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" event={"ID":"e454edad-5cc3-4ad5-99b5-fde52146d6ed","Type":"ContainerDied","Data":"7b2e4271cae2faa377277d7d7770c19a7ea09fe62b18d66a6579f6ef0ccd8207"} Oct 30 21:44:02.908019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:02.907937 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podStartSLOduration=3.907914732 podStartE2EDuration="3.907914732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:02.907636154 +0000 UTC m=+905.103327273" watchObservedRunningTime="2026-10-30 21:44:02.907914732 +0000 UTC m=+905.103605852" Oct 30 21:44:03.038128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.029016 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:03.053812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.053784 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:03.507060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.476266 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:44:03.709131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.661197 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.716542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.716471 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:03.716696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.716618 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.716696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.716675 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.716857 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.716829 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.717023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.717000 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.717244 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.717212 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.717323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.717248 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.717323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.717303 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.717428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.717380 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert\") pod \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\" (UID: \"e454edad-5cc3-4ad5-99b5-fde52146d6ed\") " Oct 30 21:44:03.718485 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.718434 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.718747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.718726 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:03.720781 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.720753 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:03.721220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.721192 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:03.721818 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.721773 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:03.737662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.737587 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:03.739518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.739394 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:03.799304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.799261 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:03.857378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.824535 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.857378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.824574 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.857378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.824588 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.857378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.824605 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.905983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.903253 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.905983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.903293 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.905983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.903312 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e454edad-5cc3-4ad5-99b5-fde52146d6ed-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.905983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.873224 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm" pod "e454edad-5cc3-4ad5-99b5-fde52146d6ed" (UID: "e454edad-5cc3-4ad5-99b5-fde52146d6ed"). InnerVolumeSpecName "kube-api-access-zvrkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:03.982756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.982480 6808 generic.go:356] "Generic (PLEG): container finished" podID="758b393e-7ff8-4e4b-a608-3e1288b3dc21" containerID="e1d5336aa74c84facfd0c6765f867ad310e2017bade5172546d6038e395f6663" exitCode=0 Oct 30 21:44:03.982756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.982687 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" event={"ID":"758b393e-7ff8-4e4b-a608-3e1288b3dc21","Type":"ContainerDied","Data":"e1d5336aa74c84facfd0c6765f867ad310e2017bade5172546d6038e395f6663"} Oct 30 21:44:03.993894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.993858 6808 generic.go:356] "Generic (PLEG): container finished" podID="bd52cd12-8c2e-490c-af55-fbb9596d22dc" containerID="dd758a024df872e8d05be87b8a5902205a3ddfc4d14f57493d6bf1d68f89193a" exitCode=0 Oct 30 21:44:03.994109 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.994013 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" event={"ID":"bd52cd12-8c2e-490c-af55-fbb9596d22dc","Type":"ContainerDied","Data":"dd758a024df872e8d05be87b8a5902205a3ddfc4d14f57493d6bf1d68f89193a"} Oct 30 21:44:03.997985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.997940 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" Oct 30 21:44:04.000122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.998993 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs" event={"ID":"e454edad-5cc3-4ad5-99b5-fde52146d6ed","Type":"ContainerDied","Data":"2df0207be0bb195b612c50c88f8f2a08e7f4bbab8c41e627a4eb80d2e597bf6e"} Oct 30 21:44:04.000122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.999049 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:04.000122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:03.999077 6808 scope.go:122] "RemoveContainer" containerID="7b2e4271cae2faa377277d7d7770c19a7ea09fe62b18d66a6579f6ef0ccd8207" Oct 30 21:44:04.000658 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.000157 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:04.007302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.007054 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrkm\" (UniqueName: \"kubernetes.io/projected/e454edad-5cc3-4ad5-99b5-fde52146d6ed-kube-api-access-zvrkm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:04.019493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.019437 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:44:04.108242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.108209 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l\" (UniqueName: \"kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l\") pod \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " Oct 30 21:44:04.108505 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.108484 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume\") pod \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\" (UID: \"bd52cd12-8c2e-490c-af55-fbb9596d22dc\") " Oct 30 21:44:04.120206 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.120172 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume" pod "bd52cd12-8c2e-490c-af55-fbb9596d22dc" (UID: "bd52cd12-8c2e-490c-af55-fbb9596d22dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:04.126077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.126049 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67"] Oct 30 21:44:04.128225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.126495 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" podUID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" containerName="istio-proxy" containerID="cri-o://ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843" gracePeriod=30 Oct 30 21:44:04.129279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.129253 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l" pod "bd52cd12-8c2e-490c-af55-fbb9596d22dc" (UID: "bd52cd12-8c2e-490c-af55-fbb9596d22dc"). InnerVolumeSpecName "kube-api-access-m999l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:04.134418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.134396 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs"] Oct 30 21:44:04.163027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.144850 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a5af9561-openshift-default-79958dfb77-g6lzs"] Oct 30 21:44:04.182847 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.182800 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:44:04.276988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.276885 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m999l\" (UniqueName: \"kubernetes.io/projected/bd52cd12-8c2e-490c-af55-fbb9596d22dc-kube-api-access-m999l\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:04.277194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.277174 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd52cd12-8c2e-490c-af55-fbb9596d22dc-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:04.460978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.442999 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p\" (UniqueName: \"kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p\") pod \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " Oct 30 21:44:04.460978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.443180 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume\") pod \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\" (UID: \"758b393e-7ff8-4e4b-a608-3e1288b3dc21\") " Oct 30 21:44:04.461520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.461463 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume" pod "758b393e-7ff8-4e4b-a608-3e1288b3dc21" (UID: "758b393e-7ff8-4e4b-a608-3e1288b3dc21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:04.565521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.565425 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b393e-7ff8-4e4b-a608-3e1288b3dc21-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:04.644226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.644187 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p" pod "758b393e-7ff8-4e4b-a608-3e1288b3dc21" (UID: "758b393e-7ff8-4e4b-a608-3e1288b3dc21"). InnerVolumeSpecName "kube-api-access-zlt4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:04.675542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:04.675499 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlt4p\" (UniqueName: \"kubernetes.io/projected/758b393e-7ff8-4e4b-a608-3e1288b3dc21-kube-api-access-zlt4p\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.005943 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.005918 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" Oct 30 21:44:05.006438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.005976 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl" event={"ID":"758b393e-7ff8-4e4b-a608-3e1288b3dc21","Type":"ContainerDied","Data":"4be00750de34e97d74b52c78b4fc86b3815c0be3d55dd0c63a9567bc68a50118"} Oct 30 21:44:05.006438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.006034 6808 scope.go:122] "RemoveContainer" containerID="e1d5336aa74c84facfd0c6765f867ad310e2017bade5172546d6038e395f6663" Oct 30 21:44:05.007689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.007656 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" Oct 30 21:44:05.007689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.007663 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z" event={"ID":"bd52cd12-8c2e-490c-af55-fbb9596d22dc","Type":"ContainerDied","Data":"1c4f37b5e59b85fd79e0d512ed35f86092a1694920d094dd239f59c6b97c95b4"} Oct 30 21:44:05.019409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.019379 6808 scope.go:122] "RemoveContainer" containerID="dd758a024df872e8d05be87b8a5902205a3ddfc4d14f57493d6bf1d68f89193a" Oct 30 21:44:05.060821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.060786 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z"] Oct 30 21:44:05.068280 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.068228 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d718596-6c79b9b747-2j55z"] Oct 30 21:44:05.089984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.089918 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl"] Oct 30 21:44:05.096110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.096085 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d718596-776fb7767-vnzfl"] Oct 30 21:44:05.504166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.504138 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9"] Oct 30 21:44:05.505153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505131 6808 cpu_manager.go:414] "containerMap: removing container" podUID="758b393e-7ff8-4e4b-a608-3e1288b3dc21" containerName="payload-pre-processing" Oct 30 21:44:05.505276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505265 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758b393e-7ff8-4e4b-a608-3e1288b3dc21" containerName="payload-pre-processing" Oct 30 21:44:05.505355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505348 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bd52cd12-8c2e-490c-af55-fbb9596d22dc" containerName="payload-processing" Oct 30 21:44:05.505404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505397 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd52cd12-8c2e-490c-af55-fbb9596d22dc" containerName="payload-processing" Oct 30 21:44:05.505467 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505461 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" containerName="istio-proxy" Oct 30 21:44:05.505514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505508 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" containerName="istio-proxy" Oct 30 21:44:05.505700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505689 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="758b393e-7ff8-4e4b-a608-3e1288b3dc21" containerName="payload-pre-processing" Oct 30 21:44:05.505776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505769 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd52cd12-8c2e-490c-af55-fbb9596d22dc" containerName="payload-processing" Oct 30 21:44:05.505831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.505824 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" containerName="istio-proxy" Oct 30 21:44:05.513503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.513479 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.524480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.524435 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1f725d-openshift-default-dockercfg-lmcqc\"" Oct 30 21:44:05.534647 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.534618 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9"] Oct 30 21:44:05.607237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607069 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607121 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607190 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607229 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607257 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607299 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607399 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607426 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.607619 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.607593 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffz5v\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.710638 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.710699 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.710735 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.710778 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.710882 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.710907 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.711092 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffz5v\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.711249 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.712981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.711284 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.738175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.736370 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.738175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.736730 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.738175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.737566 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.738175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.737867 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.750994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.744060 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.781503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.781393 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.792103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.792034 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.810930 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.810862 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:05.852692 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:05.844412 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffz5v\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v\") pod \"e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:06.067087 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:44:06.041093 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:44:06.067087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.044454 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758b393e-7ff8-4e4b-a608-3e1288b3dc21" path="/var/lib/kubelet/pods/758b393e-7ff8-4e4b-a608-3e1288b3dc21/volumes" Oct 30 21:44:06.067087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.045348 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd52cd12-8c2e-490c-af55-fbb9596d22dc" path="/var/lib/kubelet/pods/bd52cd12-8c2e-490c-af55-fbb9596d22dc/volumes" Oct 30 21:44:06.080688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.061053 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e454edad-5cc3-4ad5-99b5-fde52146d6ed" path="/var/lib/kubelet/pods/e454edad-5cc3-4ad5-99b5-fde52146d6ed/volumes" Oct 30 21:44:06.148359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.148030 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:06.534014 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:44:06.533984 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd168855_dfa6_412d_a18b_03a29a6bb7ca.slice/crio-25856bdeb1df7475e188c1225e8d932a19ff594442dadd7d5648eba3f0bee595 WatchSource:0}: Error finding container 25856bdeb1df7475e188c1225e8d932a19ff594442dadd7d5648eba3f0bee595: Status 404 returned error can't find the container with id 25856bdeb1df7475e188c1225e8d932a19ff594442dadd7d5648eba3f0bee595 Oct 30 21:44:06.534820 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.534791 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9"] Oct 30 21:44:06.537512 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.537475 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:06.537625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.537555 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:06.537625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.537595 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:06.580168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.580134 6808 patch_prober.go:29] interesting pod/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:44:06.580392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.580365 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" podUID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:06.952983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.948917 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz"] Oct 30 21:44:06.991178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.966177 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:06.991178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.986356 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2b36c1-dockercfg-qftlt\"" Oct 30 21:44:06.991178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.986422 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2b36c1\"" Oct 30 21:44:06.991178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.966086 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn"] Oct 30 21:44:06.999414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.997859 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:06.999414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:06.998038 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhhf\" (UniqueName: \"kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf\") pod \"payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:07.007959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.001245 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz"] Oct 30 21:44:07.007959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.001287 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn"] Oct 30 21:44:07.007959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.001384 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.106010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.102978 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhhf\" (UniqueName: \"kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf\") pod \"payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:07.106010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.103257 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:07.106010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.104429 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:07.279030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.224141 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jx6s\" (UniqueName: \"kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s\") pod \"payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.286089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.279457 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.363975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.349165 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" event={"ID":"cd168855-dfa6-412d-a18b-03a29a6bb7ca","Type":"ContainerStarted","Data":"c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1"} Oct 30 21:44:07.363975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.349234 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" event={"ID":"cd168855-dfa6-412d-a18b-03a29a6bb7ca","Type":"ContainerStarted","Data":"25856bdeb1df7475e188c1225e8d932a19ff594442dadd7d5648eba3f0bee595"} Oct 30 21:44:07.403224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.397841 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhhf\" (UniqueName: \"kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf\") pod \"payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:07.421012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.411889 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podStartSLOduration=2.411867624 podStartE2EDuration="2.411867624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:07.384215865 +0000 UTC m=+909.579906985" watchObservedRunningTime="2026-10-30 21:44:07.411867624 +0000 UTC m=+909.607558744" Oct 30 21:44:07.425253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.425197 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jx6s\" (UniqueName: \"kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s\") pod \"payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.425635 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.425592 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.427088 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.427045 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.480979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.478750 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jx6s\" (UniqueName: \"kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s\") pod \"payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:07.643042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.642974 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:07.747970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:07.725563 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:08.164418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.160508 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:08.184993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.177887 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:08.215105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.215074 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn"] Oct 30 21:44:08.338283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.338253 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz"] Oct 30 21:44:08.378138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.377034 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" event={"ID":"815fbaae-7bbc-4560-928b-269a8b34634f","Type":"ContainerStarted","Data":"671581d11695ce5aa0ca65a75fff1b8822b8337fb20920afc498541b26b0c546"} Oct 30 21:44:08.378138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.377107 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" event={"ID":"815fbaae-7bbc-4560-928b-269a8b34634f","Type":"ContainerStarted","Data":"5acda97de50a6a8964ad1d73a97c7f5dc2a0b88cf02272f659bba25ba62a93dd"} Oct 30 21:44:08.378138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.378130 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:08.381142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.381109 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" event={"ID":"a2e60288-7a9a-449f-85c7-897f9b5a1b99","Type":"ContainerStarted","Data":"05372b7f59bc1fc426e00cd00999df9352d7b86d9e962bd5329dc0a9d7446c19"} Oct 30 21:44:08.381471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.381452 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:08.383315 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.383294 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:44:08.406987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:08.403338 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" podStartSLOduration=2.403320801 podStartE2EDuration="2.403320801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:08.400716716 +0000 UTC m=+910.596407837" watchObservedRunningTime="2026-10-30 21:44:08.403320801 +0000 UTC m=+910.599011916" Oct 30 21:44:09.393340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:09.393298 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" event={"ID":"a2e60288-7a9a-449f-85c7-897f9b5a1b99","Type":"ContainerStarted","Data":"a2e1f1ff199e1c5513abf96cb33ed32005b80ce7ebe4d895c0a7673ab17d670f"} Oct 30 21:44:09.393826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:09.393692 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:09.419811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:09.419749 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" podStartSLOduration=3.419732566 podStartE2EDuration="3.419732566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:09.416736071 +0000 UTC m=+911.612427216" watchObservedRunningTime="2026-10-30 21:44:09.419732566 +0000 UTC m=+911.615423682" Oct 30 21:44:11.341766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.341739 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:44:11.422678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.422643 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d"] Oct 30 21:44:11.423465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.423442 6808 cpu_manager.go:414] "containerMap: removing container" podUID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" containerName="istio-proxy" Oct 30 21:44:11.423465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.423464 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" containerName="istio-proxy" Oct 30 21:44:11.423673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.423658 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" containerName="istio-proxy" Oct 30 21:44:11.427180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.427160 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.438976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.438941 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1f725d-dockercfg-fh48c\"" Oct 30 21:44:11.440562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.440289 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1f725d\"" Oct 30 21:44:11.445246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.445223 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d"] Oct 30 21:44:11.455279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.455254 6808 generic.go:356] "Generic (PLEG): container finished" podID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" containerID="ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843" exitCode=0 Oct 30 21:44:11.455413 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.455320 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" Oct 30 21:44:11.455413 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.455344 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" event={"ID":"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5","Type":"ContainerDied","Data":"ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843"} Oct 30 21:44:11.455413 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.455399 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67" event={"ID":"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5","Type":"ContainerDied","Data":"bf48e31474561ada013a9c344be45323478f87ddc09054d8f36d3c376165a862"} Oct 30 21:44:11.455591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.455429 6808 scope.go:122] "RemoveContainer" containerID="ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843" Oct 30 21:44:11.470916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.470895 6808 scope.go:122] "RemoveContainer" containerID="ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843" Oct 30 21:44:11.471322 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:44:11.471297 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843\": container with ID starting with ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843 not found: ID does not exist" containerID="ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843" Oct 30 21:44:11.471473 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.471328 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843"} err="failed to get container status \"ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843\": rpc error: code = NotFound desc = could not find container \"ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843\": container with ID starting with ebf23ef088b432ba0d45834c378883106b6fef4b19058055870d717c55c8e843 not found: ID does not exist" Oct 30 21:44:11.485593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.485570 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.485713 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.485672 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.485779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.485736 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.485827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.485798 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.485873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.485829 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.485923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.485886 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.486060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486041 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.486223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486196 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.486503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486243 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:11.486598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486267 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:11.486598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486419 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:11.486598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486479 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs\") pod \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\" (UID: \"0ab1d9a8-9bae-483f-9dec-e23c176a6ed5\") " Oct 30 21:44:11.486598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486557 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:11.486865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486838 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.486996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.486975 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbckr\" (UniqueName: \"kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr\") pod \"payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.487238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.487218 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:11.487576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.487552 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.487663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.487579 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.487663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.487598 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.487663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.487612 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.487663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.487625 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.488161 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.488135 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:11.488264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.488235 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:11.488587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.488557 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "kube-api-access-p8rqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:11.488679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.488649 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy" pod "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" (UID: "0ab1d9a8-9bae-483f-9dec-e23c176a6ed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:11.594020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.592115 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.594020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.592237 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbckr\" (UniqueName: \"kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr\") pod \"payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.594020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.592544 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.594020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.592562 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.594020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.592578 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rqh\" (UniqueName: \"kubernetes.io/projected/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-kube-api-access-p8rqh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.594020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.592594 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:11.628700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.608467 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.663974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.638435 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbckr\" (UniqueName: \"kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr\") pod \"payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.771057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.756207 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:11.918637 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:11.918540 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67"] Oct 30 21:44:12.017079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:12.016266 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6d718596-openshift-default-8c4fdb8b9-82c67"] Oct 30 21:44:12.199267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:12.195974 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d"] Oct 30 21:44:12.465383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:12.465292 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" event={"ID":"9b7dfc83-82fe-43b7-af39-bcad0c9a7957","Type":"ContainerStarted","Data":"310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65"} Oct 30 21:44:12.465383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:12.465352 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" event={"ID":"9b7dfc83-82fe-43b7-af39-bcad0c9a7957","Type":"ContainerStarted","Data":"fbaa6387060826259a4d2885be9e7f3964371aefb54193abf7d39f57366bac2e"} Oct 30 21:44:12.465383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:12.465382 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:12.487135 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:12.487088 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" podStartSLOduration=1.487075824 podStartE2EDuration="1.487075824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:12.483762199 +0000 UTC m=+914.679453330" watchObservedRunningTime="2026-10-30 21:44:12.487075824 +0000 UTC m=+914.682766943" Oct 30 21:44:13.931214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:13.931181 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab1d9a8-9bae-483f-9dec-e23c176a6ed5" path="/var/lib/kubelet/pods/0ab1d9a8-9bae-483f-9dec-e23c176a6ed5/volumes" Oct 30 21:44:18.656703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:18.656667 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc"] Oct 30 21:44:18.657492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:18.657457 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" podUID="2c2b22dd-a079-464f-8a55-8f782061df28" containerName="istio-proxy" containerID="cri-o://2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76" gracePeriod=30 Oct 30 21:44:20.045423 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.044431 6808 patch_prober.go:29] interesting pod/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:44:20.045423 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.044508 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" podUID="2c2b22dd-a079-464f-8a55-8f782061df28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:44:20.319609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.319527 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx"] Oct 30 21:44:20.339375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.339347 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx"] Oct 30 21:44:20.339680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.339662 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.350827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.348607 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-0b412a-openshift-default-dockercfg-264dc\"" Oct 30 21:44:20.468077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.467894 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468354 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468425 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468546 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkk7\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468649 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468689 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468717 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468811 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.469163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.468873 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.480817 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.474713 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:44:20.506498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.481152 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:44:20.581026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.577663 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.577738 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.577795 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.577901 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkk7\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.580817 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.580997 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.581056 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.581098 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.581242 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.581466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.581352 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.608653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.582273 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.608653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.582787 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.608653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.583607 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.608653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.585240 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.611824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.611794 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.660638 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.660571 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.671300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.671267 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkk7\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.701759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.701691 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo\") pod \"e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:20.971665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:20.970254 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:21.565394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:21.565042 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx"] Oct 30 21:44:21.571992 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:44:21.571942 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb6ce41_4a25_40f7_8a38_bdfb625c9653.slice/crio-dc25d59cf50b3ae4612783fd2dea0daf67b10f4382ddcbacbf157fa66f300fa6 WatchSource:0}: Error finding container dc25d59cf50b3ae4612783fd2dea0daf67b10f4382ddcbacbf157fa66f300fa6: Status 404 returned error can't find the container with id dc25d59cf50b3ae4612783fd2dea0daf67b10f4382ddcbacbf157fa66f300fa6 Oct 30 21:44:21.575426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:21.575391 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:21.575531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:21.575474 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:21.575531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:21.575520 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:21.599939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:21.599913 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" event={"ID":"6bb6ce41-4a25-40f7-8a38-bdfb625c9653","Type":"ContainerStarted","Data":"dc25d59cf50b3ae4612783fd2dea0daf67b10f4382ddcbacbf157fa66f300fa6"} Oct 30 21:44:22.608897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:22.608854 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" event={"ID":"6bb6ce41-4a25-40f7-8a38-bdfb625c9653","Type":"ContainerStarted","Data":"3808aa0bc3d082129ed61048ccff039d8d91e8f946cc1180ac20340ccf13477b"} Oct 30 21:44:22.646791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:22.646743 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" podStartSLOduration=2.646730518 podStartE2EDuration="2.646730518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:22.646159498 +0000 UTC m=+924.841850623" watchObservedRunningTime="2026-10-30 21:44:22.646730518 +0000 UTC m=+924.842421637" Oct 30 21:44:22.971761 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:22.971725 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:22.981005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:22.980918 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:23.476875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:23.476847 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:44:23.617728 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:23.617700 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:23.635332 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:23.635300 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:44:24.121086 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.121061 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:44:24.260146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260117 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260348 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260233 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260348 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260290 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260349 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260402 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260438 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:24.260640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260481 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260570 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260578 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:24.260640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260598 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260704 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs\") pod \"2c2b22dd-a079-464f-8a55-8f782061df28\" (UID: \"2c2b22dd-a079-464f-8a55-8f782061df28\") " Oct 30 21:44:24.260849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260804 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:24.260996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.260851 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:24.261424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.261395 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:24.261523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.261498 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.261523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.261523 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.261650 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.261541 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2b22dd-a079-464f-8a55-8f782061df28-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.261650 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.261557 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.262754 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.262732 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "kube-api-access-zn66r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:24.262937 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.262902 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:24.263268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.263247 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:24.263415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.263397 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo" pod "2c2b22dd-a079-464f-8a55-8f782061df28" (UID: "2c2b22dd-a079-464f-8a55-8f782061df28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:24.363591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.363503 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2b22dd-a079-464f-8a55-8f782061df28-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.363591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.363535 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.363591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.363546 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2b22dd-a079-464f-8a55-8f782061df28-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.363591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.363556 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn66r\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-kube-api-access-zn66r\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.363591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.363567 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2b22dd-a079-464f-8a55-8f782061df28-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:24.624119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.624034 6808 generic.go:356] "Generic (PLEG): container finished" podID="2c2b22dd-a079-464f-8a55-8f782061df28" containerID="2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76" exitCode=0 Oct 30 21:44:24.624119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.624087 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" Oct 30 21:44:24.624581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.624115 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" event={"ID":"2c2b22dd-a079-464f-8a55-8f782061df28","Type":"ContainerDied","Data":"2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76"} Oct 30 21:44:24.624581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.624171 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc" event={"ID":"2c2b22dd-a079-464f-8a55-8f782061df28","Type":"ContainerDied","Data":"0b50b1768ee48d363efa5849897fa98ea10c6bd1ff123e091c2ebd1eab25656b"} Oct 30 21:44:24.624581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.624192 6808 scope.go:122] "RemoveContainer" containerID="2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76" Oct 30 21:44:24.635560 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.635544 6808 scope.go:122] "RemoveContainer" containerID="2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76" Oct 30 21:44:24.635856 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:44:24.635832 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76\": container with ID starting with 2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76 not found: ID does not exist" containerID="2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76" Oct 30 21:44:24.635943 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.635862 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76"} err="failed to get container status \"2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76\": rpc error: code = NotFound desc = could not find container \"2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76\": container with ID starting with 2bd4928dcbd22332c037075c866bc65413345f29270779eec910f736db8c4b76 not found: ID does not exist" Oct 30 21:44:24.665361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.665332 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc"] Oct 30 21:44:24.668701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:24.668678 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-484f7514-openshift-default-95f65bdb7-mlksc"] Oct 30 21:44:25.930029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:25.929994 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2b22dd-a079-464f-8a55-8f782061df28" path="/var/lib/kubelet/pods/2c2b22dd-a079-464f-8a55-8f782061df28/volumes" Oct 30 21:44:26.396125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.396092 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t"] Oct 30 21:44:26.396805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.396790 6808 cpu_manager.go:414] "containerMap: removing container" podUID="2c2b22dd-a079-464f-8a55-8f782061df28" containerName="istio-proxy" Oct 30 21:44:26.396857 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.396806 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2b22dd-a079-464f-8a55-8f782061df28" containerName="istio-proxy" Oct 30 21:44:26.396933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.396924 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2b22dd-a079-464f-8a55-8f782061df28" containerName="istio-proxy" Oct 30 21:44:26.401811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.401790 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.408968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.408858 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-0b412a\"" Oct 30 21:44:26.409335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.409307 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-0b412a-dockercfg-f2tq4\"" Oct 30 21:44:26.439686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.439647 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t"] Oct 30 21:44:26.446730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.446702 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k"] Oct 30 21:44:26.459424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.459240 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.468039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.467671 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k"] Oct 30 21:44:26.509551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.509518 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxv8\" (UniqueName: \"kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8\") pod \"payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.509738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.509581 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.509738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.509664 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkhj\" (UniqueName: \"kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj\") pod \"payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.509875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.509741 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.611297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.611264 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkhj\" (UniqueName: \"kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj\") pod \"payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.611548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.611336 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.611647 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.611611 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxv8\" (UniqueName: \"kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8\") pod \"payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.611702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.611663 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.612367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.612340 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.612573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.612553 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.630138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.630109 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxv8\" (UniqueName: \"kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8\") pod \"payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:26.638797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.638765 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkhj\" (UniqueName: \"kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj\") pod \"payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.751021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.750041 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:26.836129 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:26.836095 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:27.658887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:27.626906 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k"] Oct 30 21:44:27.703325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:27.701730 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t"] Oct 30 21:44:27.794170 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:27.793411 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" event={"ID":"6017d829-5ae1-43ae-a65e-9bb1c856db13","Type":"ContainerStarted","Data":"deaa4ee989e40f35a3d16e7bd667b4c06d0cdb0a3a73b9b79371312b82e53760"} Oct 30 21:44:28.803092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.803057 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" event={"ID":"adbedada-9cf6-4d94-9edb-0f9f92010d5c","Type":"ContainerStarted","Data":"025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7"} Oct 30 21:44:28.803651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.803624 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:28.803803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.803662 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" event={"ID":"adbedada-9cf6-4d94-9edb-0f9f92010d5c","Type":"ContainerStarted","Data":"dd177d6d9b8b2286a90e281f26a42c225801fa96509244b6bc43ec896175987c"} Oct 30 21:44:28.805032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.805007 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" event={"ID":"6017d829-5ae1-43ae-a65e-9bb1c856db13","Type":"ContainerStarted","Data":"9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920"} Oct 30 21:44:28.805144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.805126 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:28.823912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.823863 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" podStartSLOduration=2.823847421 podStartE2EDuration="2.823847421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:28.823128236 +0000 UTC m=+931.018819353" watchObservedRunningTime="2026-10-30 21:44:28.823847421 +0000 UTC m=+931.019538541" Oct 30 21:44:28.845381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:28.845329 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" podStartSLOduration=2.845313005 podStartE2EDuration="2.845313005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:28.841457232 +0000 UTC m=+931.037148354" watchObservedRunningTime="2026-10-30 21:44:28.845313005 +0000 UTC m=+931.041004125" Oct 30 21:44:29.072321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:29.072225 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k"] Oct 30 21:44:29.087284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:29.087250 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc981ee-gcm5k"] Oct 30 21:44:29.930043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:29.930010 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e8d17e-eadc-49d8-8219-4ebbc729e78e" path="/var/lib/kubelet/pods/e1e8d17e-eadc-49d8-8219-4ebbc729e78e/volumes" Oct 30 21:44:31.552764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.552726 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh"] Oct 30 21:44:31.557398 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.557377 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz"] Oct 30 21:44:31.557743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.557518 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.562105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.562084 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:31.568502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.568475 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-dc3edf\"" Oct 30 21:44:31.568777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.568758 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-dc3edf-dockercfg-pksjh\"" Oct 30 21:44:31.570358 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.570336 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh"] Oct 30 21:44:31.574610 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.574584 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz"] Oct 30 21:44:31.598749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.598718 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.598931 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.598913 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4cc\" (UniqueName: \"kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc\") pod \"payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.714030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.706033 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.714030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.706267 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4cc\" (UniqueName: \"kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc\") pod \"payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.714030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.707662 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.752409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.742921 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4cc\" (UniqueName: \"kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc\") pod \"payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.814018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.813263 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:31.814018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.813567 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8skx\" (UniqueName: \"kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx\") pod \"payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:31.886406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.885219 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:31.929219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.929175 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:31.929459 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.929394 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8skx\" (UniqueName: \"kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx\") pod \"payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:31.930868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.930839 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:31.965659 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:31.965611 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8skx\" (UniqueName: \"kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx\") pod \"payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:32.198549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.198233 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:32.400598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.400569 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh"] Oct 30 21:44:32.401247 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:44:32.401215 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cadc6b8_d1c3_4156_8970_ecfc72bebfc5.slice/crio-f06b42ef56ff92ef471cd41f55fd2140c8752c0ddc225051f3cc2bb6d60dd399 WatchSource:0}: Error finding container f06b42ef56ff92ef471cd41f55fd2140c8752c0ddc225051f3cc2bb6d60dd399: Status 404 returned error can't find the container with id f06b42ef56ff92ef471cd41f55fd2140c8752c0ddc225051f3cc2bb6d60dd399 Oct 30 21:44:32.609022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.608897 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz"] Oct 30 21:44:32.612170 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:44:32.612145 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f1c4ee_9b0d_4283_ac84_77d7f209eb14.slice/crio-dc302e489fa02c4b9cc21a27ccc39b0ebd84d391c90c8a01bad03d08b91cc06b WatchSource:0}: Error finding container dc302e489fa02c4b9cc21a27ccc39b0ebd84d391c90c8a01bad03d08b91cc06b: Status 404 returned error can't find the container with id dc302e489fa02c4b9cc21a27ccc39b0ebd84d391c90c8a01bad03d08b91cc06b Oct 30 21:44:32.869235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.869096 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" event={"ID":"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5","Type":"ContainerStarted","Data":"2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe"} Oct 30 21:44:32.869235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.869170 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" event={"ID":"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5","Type":"ContainerStarted","Data":"f06b42ef56ff92ef471cd41f55fd2140c8752c0ddc225051f3cc2bb6d60dd399"} Oct 30 21:44:32.869235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.869203 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:32.871451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.871420 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" event={"ID":"70f1c4ee-9b0d-4283-ac84-77d7f209eb14","Type":"ContainerStarted","Data":"9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d"} Oct 30 21:44:32.871603 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.871464 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" event={"ID":"70f1c4ee-9b0d-4283-ac84-77d7f209eb14","Type":"ContainerStarted","Data":"dc302e489fa02c4b9cc21a27ccc39b0ebd84d391c90c8a01bad03d08b91cc06b"} Oct 30 21:44:32.871603 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.871564 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:32.893253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.893196 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" podStartSLOduration=1.893183702 podStartE2EDuration="1.893183702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:32.88943782 +0000 UTC m=+935.085128938" watchObservedRunningTime="2026-10-30 21:44:32.893183702 +0000 UTC m=+935.088874844" Oct 30 21:44:32.914966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:32.912778 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" podStartSLOduration=1.912764396 podStartE2EDuration="1.912764396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:32.909231763 +0000 UTC m=+935.104922883" watchObservedRunningTime="2026-10-30 21:44:32.912764396 +0000 UTC m=+935.108455516" Oct 30 21:44:35.576029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.540459 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s"] Oct 30 21:44:35.586403 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.586372 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s"] Oct 30 21:44:35.586564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.586491 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.591916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.591887 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-0b412a-serving-cert\"" Oct 30 21:44:35.828011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.815136 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.828011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.815230 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.828011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.815524 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.828011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.815782 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.970985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.966322 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.970985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.966530 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.970985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.966613 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.970985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.966699 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.970985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:35.969616 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:35.994994 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:44:35.993797 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:44:36.026730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:36.026664 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:36.027509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:36.027445 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:36.044973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:36.039441 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5\") pod \"maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:36.209973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:36.209809 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:37.608268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:37.608239 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s"] Oct 30 21:44:37.608885 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:44:37.608857 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27aea3b1_9630_4c78_9ae5_10fbd1718ec0.slice/crio-b57a063aa79e4ab346ccd53401a9edc6007f99fa980442fd1c4858ed37cf728b WatchSource:0}: Error finding container b57a063aa79e4ab346ccd53401a9edc6007f99fa980442fd1c4858ed37cf728b: Status 404 returned error can't find the container with id b57a063aa79e4ab346ccd53401a9edc6007f99fa980442fd1c4858ed37cf728b Oct 30 21:44:38.096081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:38.096042 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" event={"ID":"27aea3b1-9630-4c78-9ae5-10fbd1718ec0","Type":"ContainerStarted","Data":"b57a063aa79e4ab346ccd53401a9edc6007f99fa980442fd1c4858ed37cf728b"} Oct 30 21:44:39.107682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:39.107407 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" event={"ID":"27aea3b1-9630-4c78-9ae5-10fbd1718ec0","Type":"ContainerStarted","Data":"bef514d5c9c8f9fc287d22ddef6aa158b9fbe03547eed36976c709939641feae"} Oct 30 21:44:39.107682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:39.107631 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:39.136182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:39.135835 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" podStartSLOduration=3.568832685 podStartE2EDuration="4.135823568s" totalImagesPullingTime="566.990883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:39.133970278 +0000 UTC m=+941.329661390" watchObservedRunningTime="2026-10-30 21:44:39.135823568 +0000 UTC m=+941.331514724" Oct 30 21:44:39.816862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:39.816836 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:44:39.817079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:39.816936 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:44:40.549801 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.549769 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr"] Oct 30 21:44:40.594021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.586939 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.596283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.596246 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-dc3edf-serving-cert\"" Oct 30 21:44:40.603113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.603088 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr"] Oct 30 21:44:40.720869 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.720815 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.721279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.721253 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.721568 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.721548 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj5t\" (UniqueName: \"kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.721754 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.721738 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.828111 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.828002 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj5t\" (UniqueName: \"kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.828391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.828370 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.828764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.828743 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.829092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.829035 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.841983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.829764 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.889033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.843965 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.889033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.882584 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.913982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.912089 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj5t\" (UniqueName: \"kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t\") pod \"maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:40.917981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:40.914446 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:41.687977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:41.679038 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr"] Oct 30 21:44:41.976999 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:44:41.976301 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39d8c88_44e7_428a_9425_92218b68f58e.slice/crio-e9878f5b8003459a2d722567a2093dcef02caed999df7a5a2bf1adfade89742e WatchSource:0}: Error finding container e9878f5b8003459a2d722567a2093dcef02caed999df7a5a2bf1adfade89742e: Status 404 returned error can't find the container with id e9878f5b8003459a2d722567a2093dcef02caed999df7a5a2bf1adfade89742e Oct 30 21:44:42.427560 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:42.427525 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" event={"ID":"c39d8c88-44e7-428a-9425-92218b68f58e","Type":"ContainerStarted","Data":"e9878f5b8003459a2d722567a2093dcef02caed999df7a5a2bf1adfade89742e"} Oct 30 21:44:43.437099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:43.437067 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" event={"ID":"c39d8c88-44e7-428a-9425-92218b68f58e","Type":"ContainerStarted","Data":"7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b"} Oct 30 21:44:43.437510 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:43.437219 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:43.465900 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:43.465830 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" podStartSLOduration=2.9957640789999997 podStartE2EDuration="3.465809902s" totalImagesPullingTime="470.045823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:43.461856208 +0000 UTC m=+945.657547331" watchObservedRunningTime="2026-10-30 21:44:43.465809902 +0000 UTC m=+945.661501023" Oct 30 21:44:43.882003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:43.881931 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:44:43.882810 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:43.882787 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:44:45.181202 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:45.181170 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:44:47.482987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:47.482912 6808 generic.go:356] "Generic (PLEG): container finished" podID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerID="97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b" exitCode=0 Oct 30 21:44:47.483512 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:47.482990 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" event={"ID":"219686ae-b68b-4d6d-b32a-0de8543b2064","Type":"ContainerDied","Data":"97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b"} Oct 30 21:44:47.483512 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:47.483414 6808 scope.go:122] "RemoveContainer" containerID="97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b" Oct 30 21:44:47.485453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:47.485417 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:47.485594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:47.485494 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:47.485594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:47.485533 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:48.493226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:48.493196 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" event={"ID":"219686ae-b68b-4d6d-b32a-0de8543b2064","Type":"ContainerStarted","Data":"54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879"} Oct 30 21:44:48.997973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:48.997917 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:49.029091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:49.029060 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:49.030138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:49.030114 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:49.030254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:49.030159 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:49.471974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:49.471912 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:44:50.029828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:50.029794 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:50.030308 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:50.029855 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:51.034388 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:51.034348 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:51.034965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:51.034920 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:52.032114 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:52.032080 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:52.032420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:52.032389 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:53.031013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:53.030344 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:53.031013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:53.030409 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:54.106972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:54.102674 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.106972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:54.102749 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:55.038973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:55.032035 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:55.038973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:55.032096 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:56.033940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:56.033908 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:56.035431 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:56.035411 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:44:58.038054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.038019 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb"] Oct 30 21:44:58.044711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.044195 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.064166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.064143 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-08aafeac-openshift-default-dockercfg-qg8xv\"" Oct 30 21:44:58.069351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.069328 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb"] Oct 30 21:44:58.120492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120452 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.120862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120521 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.120862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120562 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.120862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120608 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.120862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120721 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ln5\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.120862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120765 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.121232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.120900 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.121232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.121028 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.121232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.121053 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.222507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.222440 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ln5\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.222908 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.222863 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.223307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.223264 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.223597 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.223578 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.224112 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.224081 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.224438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.224411 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.224603 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.224578 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.224790 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.224761 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.224958 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.224921 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.240065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.226398 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.240065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.226708 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.240065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.229636 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.240065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.230394 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.241111 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.241019 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.301891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.301775 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.319970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.302433 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.322534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.307618 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.322731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.322436 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ln5\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:58.372515 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:58.372477 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:44:59.154010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:59.153976 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb"] Oct 30 21:44:59.162462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:59.159919 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:59.162462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:59.160346 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:59.162462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:59.160424 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:44:59.974462 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:59.974420 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" event={"ID":"4177f737-6bc7-4647-b1d8-351e85c33cee","Type":"ContainerStarted","Data":"25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7"} Oct 30 21:44:59.974746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:44:59.974726 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" event={"ID":"4177f737-6bc7-4647-b1d8-351e85c33cee","Type":"ContainerStarted","Data":"82042d5885833de51ad27f1a8a68058e240837e8a93f9228143da1b6c92491ce"} Oct 30 21:45:00.085966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.065706 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podStartSLOduration=3.065684061 podStartE2EDuration="3.065684061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:00.045576364 +0000 UTC m=+962.241267490" watchObservedRunningTime="2026-10-30 21:45:00.065684061 +0000 UTC m=+962.261375179" Oct 30 21:45:00.226025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.212423 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b"] Oct 30 21:45:00.291375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.291339 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.293221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.293196 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk"] Oct 30 21:45:00.300061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.298229 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.300197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.300103 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml"] Oct 30 21:45:00.301591 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.301558 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:45:00.338171 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.332130 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj"] Oct 30 21:45:00.339204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.339180 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.371421 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.371340 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b"] Oct 30 21:45:00.385902 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.385871 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:45:00.386215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.386198 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml"] Oct 30 21:45:00.386346 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.374301 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.403615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403439 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmhq\" (UniqueName: \"kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.403615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403499 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-ss6qk\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.403615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403537 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhzjz\" (UniqueName: \"kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz\") pod \"maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.403615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403574 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqw8f\" (UniqueName: \"kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f\") pod \"maas-api-key-cleanup-29889945-ss6qk\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.403942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403762 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8sn\" (UniqueName: \"kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.403942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403802 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.403942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.403836 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.404160 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.404021 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.418689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.418658 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk"] Oct 30 21:45:00.422002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.421749 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj"] Oct 30 21:45:00.424249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.424227 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:45:00.505998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.505906 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8sn\" (UniqueName: \"kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.505998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.505970 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.506221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.506013 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.506221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.506168 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.506377 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.506345 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmhq\" (UniqueName: \"kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.506439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.506389 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-ss6qk\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.506439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.506420 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhzjz\" (UniqueName: \"kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz\") pod \"maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.506547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.506447 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqw8f\" (UniqueName: \"kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f\") pod \"maas-api-key-cleanup-29889945-ss6qk\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.507433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.507013 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.507433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.507046 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.507433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.507353 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.507686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.507566 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-ss6qk\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.518967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.518925 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8sn\" (UniqueName: \"kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:00.520253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.520221 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmhq\" (UniqueName: \"kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.520523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.520498 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhzjz\" (UniqueName: \"kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz\") pod \"maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.520642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.520632 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqw8f\" (UniqueName: \"kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f\") pod \"maas-api-key-cleanup-29889945-ss6qk\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.703495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.703464 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:00.746237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.746209 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:00.763374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.763301 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:00.770586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:00.770560 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:01.077292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:01.077262 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b"] Oct 30 21:45:01.087013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:01.086984 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:45:01.090378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:01.089755 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:45:01.299841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:01.299748 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk"] Oct 30 21:45:01.300617 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:45:01.300582 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f090b5_a6f1_4418_9db4_17c61ab26a73.slice/crio-c7890085d145bf9c0e457d0aac2a64e8f86e543df04de16b0a9078d9a2b21183 WatchSource:0}: Error finding container c7890085d145bf9c0e457d0aac2a64e8f86e543df04de16b0a9078d9a2b21183: Status 404 returned error can't find the container with id c7890085d145bf9c0e457d0aac2a64e8f86e543df04de16b0a9078d9a2b21183 Oct 30 21:45:01.316990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:01.316929 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml"] Oct 30 21:45:01.326735 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:01.326711 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj"] Oct 30 21:45:02.101593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.101553 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerStarted","Data":"6de793fcd42ee41c5b24692ca3487df9dd8f3af0cdff55315f5a202e9e5516f1"} Oct 30 21:45:02.101780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.101628 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerStarted","Data":"a05bda5ea158df8b27da281f2688b69c217a229347b2979d56e66e025d447e0a"} Oct 30 21:45:02.103368 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.103333 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerStarted","Data":"9654021f05f87e77abeb9fe4b89b1552c228ad625f6cebd5785594b2fcc5db0a"} Oct 30 21:45:02.103493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.103389 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerStarted","Data":"2f055ec8d34666bcd6c43b8089e53431228ee4d1b5cd3033f10b40229f405ebe"} Oct 30 21:45:02.105116 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.105088 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerStarted","Data":"c0350131e3d81136380c6070009feecc8329dbc72bf942b6083bf8cef8003168"} Oct 30 21:45:02.105281 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.105137 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerStarted","Data":"c7890085d145bf9c0e457d0aac2a64e8f86e543df04de16b0a9078d9a2b21183"} Oct 30 21:45:02.107141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.107116 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerStarted","Data":"108379e9140d29b4eec9d7b5f68e0bfad2d5b5162f651496299057dbcf4d4b49"} Oct 30 21:45:02.107264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.107153 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerStarted","Data":"2d563447915d93969c64604fe77068701b48f3b10cb8c99cdc21f4361196f394"} Oct 30 21:45:02.123629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.123587 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" podStartSLOduration=2.123571253 podStartE2EDuration="2.123571253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.120403844 +0000 UTC m=+964.316094966" watchObservedRunningTime="2026-10-30 21:45:02.123571253 +0000 UTC m=+964.319262398" Oct 30 21:45:02.172418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.172323 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" podStartSLOduration=2.172309896 podStartE2EDuration="2.172309896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.149525493 +0000 UTC m=+964.345216615" watchObservedRunningTime="2026-10-30 21:45:02.172309896 +0000 UTC m=+964.368001065" Oct 30 21:45:02.174685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.174647 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" podStartSLOduration=2.174639125 podStartE2EDuration="2.174639125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.17078082 +0000 UTC m=+964.366471939" watchObservedRunningTime="2026-10-30 21:45:02.174639125 +0000 UTC m=+964.370330304" Oct 30 21:45:02.196011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:02.195963 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" podStartSLOduration=2.195936822 podStartE2EDuration="2.195936822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.193768922 +0000 UTC m=+964.389460045" watchObservedRunningTime="2026-10-30 21:45:02.195936822 +0000 UTC m=+964.391627953" Oct 30 21:45:03.115715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.115679 6808 generic.go:356] "Generic (PLEG): container finished" podID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerID="108379e9140d29b4eec9d7b5f68e0bfad2d5b5162f651496299057dbcf4d4b49" exitCode=60 Oct 30 21:45:03.116209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.115765 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerDied","Data":"108379e9140d29b4eec9d7b5f68e0bfad2d5b5162f651496299057dbcf4d4b49"} Oct 30 21:45:03.116209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.116098 6808 scope.go:122] "RemoveContainer" containerID="108379e9140d29b4eec9d7b5f68e0bfad2d5b5162f651496299057dbcf4d4b49" Oct 30 21:45:03.117781 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.117749 6808 generic.go:356] "Generic (PLEG): container finished" podID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerID="6de793fcd42ee41c5b24692ca3487df9dd8f3af0cdff55315f5a202e9e5516f1" exitCode=60 Oct 30 21:45:03.118049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.117880 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerDied","Data":"6de793fcd42ee41c5b24692ca3487df9dd8f3af0cdff55315f5a202e9e5516f1"} Oct 30 21:45:03.118116 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.118044 6808 scope.go:122] "RemoveContainer" containerID="6de793fcd42ee41c5b24692ca3487df9dd8f3af0cdff55315f5a202e9e5516f1" Oct 30 21:45:03.119805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.119659 6808 generic.go:356] "Generic (PLEG): container finished" podID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerID="9654021f05f87e77abeb9fe4b89b1552c228ad625f6cebd5785594b2fcc5db0a" exitCode=60 Oct 30 21:45:03.119805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.119739 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerDied","Data":"9654021f05f87e77abeb9fe4b89b1552c228ad625f6cebd5785594b2fcc5db0a"} Oct 30 21:45:03.120029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.120011 6808 scope.go:122] "RemoveContainer" containerID="9654021f05f87e77abeb9fe4b89b1552c228ad625f6cebd5785594b2fcc5db0a" Oct 30 21:45:03.122104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.122078 6808 generic.go:356] "Generic (PLEG): container finished" podID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerID="c0350131e3d81136380c6070009feecc8329dbc72bf942b6083bf8cef8003168" exitCode=60 Oct 30 21:45:03.122220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.122152 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerDied","Data":"c0350131e3d81136380c6070009feecc8329dbc72bf942b6083bf8cef8003168"} Oct 30 21:45:03.122442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.122339 6808 scope.go:122] "RemoveContainer" containerID="c0350131e3d81136380c6070009feecc8329dbc72bf942b6083bf8cef8003168" Oct 30 21:45:03.776439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.776410 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw"] Oct 30 21:45:03.781900 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.781867 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:03.788481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.788452 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw"] Oct 30 21:45:03.788726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.788707 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08aafeac-dockercfg-wl4xx\"" Oct 30 21:45:03.788828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.788696 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08aafeac\"" Oct 30 21:45:03.821002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.820943 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn"] Oct 30 21:45:03.825346 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.825326 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:03.831929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.831899 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns894\" (UniqueName: \"kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:03.832642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.832231 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:03.836186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.836162 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn"] Oct 30 21:45:03.968342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.962219 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:03.968342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.962460 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns894\" (UniqueName: \"kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:03.968342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:03.963801 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:04.045500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.041971 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns894\" (UniqueName: \"kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:04.076975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.072356 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.076975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.072475 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxt8\" (UniqueName: \"kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.227991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.109680 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:04.233910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.180742 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxt8\" (UniqueName: \"kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.233910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.231421 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.233910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.232773 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerStarted","Data":"1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3"} Oct 30 21:45:04.233910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.233837 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.377137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.361753 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerStarted","Data":"895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335"} Oct 30 21:45:04.377137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.361835 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxt8\" (UniqueName: \"kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.377137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.365923 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerStarted","Data":"3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f"} Oct 30 21:45:04.377137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.371384 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerStarted","Data":"75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266"} Oct 30 21:45:04.449235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.449183 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:04.778838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.778809 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw"] Oct 30 21:45:04.924183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:04.924154 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn"] Oct 30 21:45:04.925333 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:45:04.925301 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fad2e0_8fd0_45d7_bf79_b14a4abe9f72.slice/crio-bb19dc8100fbdc7aa8f6c9651295b6d60df3fe62fb8f19e0a1e313e1d4a69932 WatchSource:0}: Error finding container bb19dc8100fbdc7aa8f6c9651295b6d60df3fe62fb8f19e0a1e313e1d4a69932: Status 404 returned error can't find the container with id bb19dc8100fbdc7aa8f6c9651295b6d60df3fe62fb8f19e0a1e313e1d4a69932 Oct 30 21:45:05.499907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.499873 6808 generic.go:356] "Generic (PLEG): container finished" podID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerID="895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335" exitCode=60 Oct 30 21:45:05.504137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.502810 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerDied","Data":"895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335"} Oct 30 21:45:05.505013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.504988 6808 scope.go:122] "RemoveContainer" containerID="9654021f05f87e77abeb9fe4b89b1552c228ad625f6cebd5785594b2fcc5db0a" Oct 30 21:45:05.513099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.512630 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" event={"ID":"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72","Type":"ContainerStarted","Data":"7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91"} Oct 30 21:45:05.513287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.513268 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" event={"ID":"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72","Type":"ContainerStarted","Data":"bb19dc8100fbdc7aa8f6c9651295b6d60df3fe62fb8f19e0a1e313e1d4a69932"} Oct 30 21:45:05.513777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.513754 6808 scope.go:122] "RemoveContainer" containerID="895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335" Oct 30 21:45:05.529504 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:05.514393 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj_odh-ai-gateway-infra(a883830b-01ec-4c82-83c1-e4635dd47f18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" Oct 30 21:45:05.561978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.513820 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:05.591985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.591919 6808 generic.go:356] "Generic (PLEG): container finished" podID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerID="3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f" exitCode=60 Oct 30 21:45:05.592379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.592342 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerDied","Data":"3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f"} Oct 30 21:45:05.593236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.593205 6808 scope.go:122] "RemoveContainer" containerID="3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f" Oct 30 21:45:05.595856 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:05.595814 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-ss6qk_odh-ai-gateway-infra(b6f090b5-a6f1-4418-9db4-17c61ab26a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" Oct 30 21:45:05.596060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.596037 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerDied","Data":"75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266"} Oct 30 21:45:05.596236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.596004 6808 generic.go:356] "Generic (PLEG): container finished" podID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerID="75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266" exitCode=60 Oct 30 21:45:05.597158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.597143 6808 scope.go:122] "RemoveContainer" containerID="75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266" Oct 30 21:45:05.597937 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:05.597918 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b_odh-ai-gateway-infra(dac3094e-c37e-40e1-943b-f05e20dd6d07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" Oct 30 21:45:05.599506 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.599483 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" event={"ID":"959951b5-b898-461e-a527-22c8f28f1b66","Type":"ContainerStarted","Data":"1895c4d1a68401dd1d867c1eae3ff57b39369ffaf172ac94cce41c86aa6942a4"} Oct 30 21:45:05.599669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.599654 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" event={"ID":"959951b5-b898-461e-a527-22c8f28f1b66","Type":"ContainerStarted","Data":"265b0439157b18db1d5ee99d0701c5f79e754b13e1502601abd37e00b9e61498"} Oct 30 21:45:05.606367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.606314 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:05.622941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.622906 6808 generic.go:356] "Generic (PLEG): container finished" podID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerID="1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3" exitCode=60 Oct 30 21:45:05.624986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.624942 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerDied","Data":"1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3"} Oct 30 21:45:05.625872 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.625849 6808 scope.go:122] "RemoveContainer" containerID="1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3" Oct 30 21:45:05.627356 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:05.627330 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml_odh-ai-gateway-infra(77252a6f-9b10-4481-b87a-4c2d9591a657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" Oct 30 21:45:05.711248 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.711218 6808 scope.go:122] "RemoveContainer" containerID="c0350131e3d81136380c6070009feecc8329dbc72bf942b6083bf8cef8003168" Oct 30 21:45:05.735650 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.735499 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" podStartSLOduration=2.7354795750000003 podStartE2EDuration="2.735479575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:05.600643701 +0000 UTC m=+967.796334822" watchObservedRunningTime="2026-10-30 21:45:05.735479575 +0000 UTC m=+967.931170696" Oct 30 21:45:05.768238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.768134 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" podStartSLOduration=2.768114631 podStartE2EDuration="2.768114631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:05.758325209 +0000 UTC m=+967.954016329" watchObservedRunningTime="2026-10-30 21:45:05.768114631 +0000 UTC m=+967.963805751" Oct 30 21:45:05.793426 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:05.793388 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:45:05.794624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.794600 6808 scope.go:122] "RemoveContainer" containerID="108379e9140d29b4eec9d7b5f68e0bfad2d5b5162f651496299057dbcf4d4b49" Oct 30 21:45:05.811614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:05.811587 6808 scope.go:122] "RemoveContainer" containerID="6de793fcd42ee41c5b24692ca3487df9dd8f3af0cdff55315f5a202e9e5516f1" Oct 30 21:45:06.635216 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:06.635186 6808 scope.go:122] "RemoveContainer" containerID="75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266" Oct 30 21:45:06.635681 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:06.635597 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b_odh-ai-gateway-infra(dac3094e-c37e-40e1-943b-f05e20dd6d07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" Oct 30 21:45:06.637125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:06.637103 6808 scope.go:122] "RemoveContainer" containerID="1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3" Oct 30 21:45:06.637507 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:06.637484 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml_odh-ai-gateway-infra(77252a6f-9b10-4481-b87a-4c2d9591a657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" Oct 30 21:45:06.639711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:06.639690 6808 scope.go:122] "RemoveContainer" containerID="895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335" Oct 30 21:45:06.640069 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:06.640037 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj_odh-ai-gateway-infra(a883830b-01ec-4c82-83c1-e4635dd47f18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" Oct 30 21:45:06.641865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:06.641846 6808 scope.go:122] "RemoveContainer" containerID="3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f" Oct 30 21:45:06.642097 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:06.642079 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-ss6qk_odh-ai-gateway-infra(b6f090b5-a6f1-4418-9db4-17c61ab26a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" Oct 30 21:45:11.740400 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.740357 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-02f6e3"] Oct 30 21:45:11.747473 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.747447 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:11.753990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.751143 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-02f6e3"] Oct 30 21:45:11.804227 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.793596 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l4j2n\"" Oct 30 21:45:11.830930 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.830720 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2xg\" (UniqueName: \"kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg\") pod \"test-curl-7342-02f6e3\" (UID: \"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd\") " pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:11.935623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.935582 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2xg\" (UniqueName: \"kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg\") pod \"test-curl-7342-02f6e3\" (UID: \"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd\") " pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:11.988253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:11.988212 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2xg\" (UniqueName: \"kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg\") pod \"test-curl-7342-02f6e3\" (UID: \"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd\") " pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:12.075786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:12.075701 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:12.357424 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:12.357394 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-02f6e3"] Oct 30 21:45:12.359288 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:45:12.359259 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc03ce16f_58c0_4a0e_9daa_c80bd8856fdd.slice/crio-e783aa332710bfeaee2ae0ca6ef9d6e45ae3345535d1e779e1fbbe86096f06c0 WatchSource:0}: Error finding container e783aa332710bfeaee2ae0ca6ef9d6e45ae3345535d1e779e1fbbe86096f06c0: Status 404 returned error can't find the container with id e783aa332710bfeaee2ae0ca6ef9d6e45ae3345535d1e779e1fbbe86096f06c0 Oct 30 21:45:12.913786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:12.913746 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-02f6e3" event={"ID":"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd","Type":"ContainerStarted","Data":"3cfb6123eae4005df3778e729280426aea514c7cea75450c4df618d58a22f6bb"} Oct 30 21:45:12.913786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:12.913789 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-02f6e3" event={"ID":"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd","Type":"ContainerStarted","Data":"e783aa332710bfeaee2ae0ca6ef9d6e45ae3345535d1e779e1fbbe86096f06c0"} Oct 30 21:45:12.931516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:12.931462 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7342-02f6e3" podStartSLOduration=1.931447195 podStartE2EDuration="1.931447195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:12.930077746 +0000 UTC m=+975.125768866" watchObservedRunningTime="2026-10-30 21:45:12.931447195 +0000 UTC m=+975.127138313" Oct 30 21:45:13.822563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:13.822522 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-02f6e3"] Oct 30 21:45:13.827025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:13.826999 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-02f6e3"] Oct 30 21:45:13.995583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:13.990155 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm"] Oct 30 21:45:14.003972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:13.996402 6808 cpu_manager.go:414] "containerMap: removing container" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" containerName="test-curl-7342-02f6e3" Oct 30 21:45:14.003972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:13.996432 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" containerName="test-curl-7342-02f6e3" Oct 30 21:45:14.003972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:13.996673 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" containerName="test-curl-7342-02f6e3" Oct 30 21:45:14.014630 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.014603 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.016223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.015661 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-137e4c"] Oct 30 21:45:14.022389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.022364 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08aafeac-serving-cert\"" Oct 30 21:45:14.057000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.056659 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-137e4c"] Oct 30 21:45:14.057000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.056691 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm"] Oct 30 21:45:14.057000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.056801 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:14.116071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.114156 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww29b\" (UniqueName: \"kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.116071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.114299 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.116071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.114331 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.116071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.114606 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.116071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.114693 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnj8\" (UniqueName: \"kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8\") pod \"test-curl-7342-137e4c\" (UID: \"bf1e6c75-adcb-42d3-a378-f3b113eb0e42\") " pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:14.218373 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.218319 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnj8\" (UniqueName: \"kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8\") pod \"test-curl-7342-137e4c\" (UID: \"bf1e6c75-adcb-42d3-a378-f3b113eb0e42\") " pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:14.218760 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.218730 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww29b\" (UniqueName: \"kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.218905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.218883 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.219047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.218919 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.219164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.219147 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.271976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.270940 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.271976 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:14.271088 6808 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-serving-cert: secret "maas-api-e2e-mt-08aafeac-serving-cert" not found Oct 30 21:45:14.271976 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:14.271153 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls podName:5e455637-ae8f-461b-8ccd-daca7dab347f nodeName:}" failed. No retries permitted until 2026-10-30 21:45:14.771133238 +0000 UTC m=+976.966824367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls") pod "maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" (UID: "5e455637-ae8f-461b-8ccd-daca7dab347f") : secret "maas-api-e2e-mt-08aafeac-serving-cert" not found Oct 30 21:45:14.319974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.306648 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.319974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.307265 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnj8\" (UniqueName: \"kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8\") pod \"test-curl-7342-137e4c\" (UID: \"bf1e6c75-adcb-42d3-a378-f3b113eb0e42\") " pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:14.356997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.349205 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww29b\" (UniqueName: \"kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.385525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.385496 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:14.853624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.853523 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.953287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.953245 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:14.959752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:14.958361 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7342-02f6e3" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" containerName="test-curl-7342-02f6e3" containerID="cri-o://3cfb6123eae4005df3778e729280426aea514c7cea75450c4df618d58a22f6bb" gracePeriod=2 Oct 30 21:45:15.359093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.358096 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:15.560419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.510473 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:45:15.639012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.638056 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:45:15.646095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.646054 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:15.714033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.710537 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-137e4c"] Oct 30 21:45:15.748013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.740235 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:15.886699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.874416 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:45:15.932694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:15.922790 6808 scope.go:122] "RemoveContainer" containerID="1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3" Oct 30 21:45:16.015682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:16.015614 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-137e4c" event={"ID":"bf1e6c75-adcb-42d3-a378-f3b113eb0e42","Type":"ContainerStarted","Data":"c9cbf3bdb37d31d23989b2f3291513a3e4f2a7c61c85749c3a2285dfe1935ddd"} Oct 30 21:45:16.733359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:16.733338 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm"] Oct 30 21:45:16.925617 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:16.925488 6808 scope.go:122] "RemoveContainer" containerID="895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335" Oct 30 21:45:17.298007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:17.297969 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" event={"ID":"5e455637-ae8f-461b-8ccd-daca7dab347f","Type":"ContainerStarted","Data":"007763ad1ebe7473092e8a720ed4691bcca68874a2ad8424413bf23051e6d641"} Oct 30 21:45:17.300843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:17.300811 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerStarted","Data":"9a19bdc0fe46f38ccf511ce6a1a736f24c114d2a9104b08eb7c3ca11ec95c9b5"} Oct 30 21:45:17.345690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:17.345504 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-137e4c" event={"ID":"bf1e6c75-adcb-42d3-a378-f3b113eb0e42","Type":"ContainerStarted","Data":"b277566711c7e9b5995cb19e7a0d7d4925dbe04bf6a01f5e031a75889773e4fb"} Oct 30 21:45:17.625969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:17.623675 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7342-137e4c" podStartSLOduration=4.62365401 podStartE2EDuration="4.62365401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:17.479260541 +0000 UTC m=+979.674951661" watchObservedRunningTime="2026-10-30 21:45:17.62365401 +0000 UTC m=+979.819345130" Oct 30 21:45:17.706358 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:17.705081 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:45:17.715076 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:17.710110 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:45:18.099942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:18.099898 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:18.162396 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:18.162316 6808 scope.go:122] "RemoveContainer" containerID="75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266" Oct 30 21:45:18.163841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:18.163347 6808 scope.go:122] "RemoveContainer" containerID="3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f" Oct 30 21:45:18.476995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:18.473873 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml"] Oct 30 21:45:18.478260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:18.477299 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:18.967859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:18.967827 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:18.996028 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:18.993884 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:19.029126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.019560 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" event={"ID":"5e455637-ae8f-461b-8ccd-daca7dab347f","Type":"ContainerStarted","Data":"ffef59ac6613bfe144944a8d60291e192880e8aa90e746df0a606f0733ba31d3"} Oct 30 21:45:19.029126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.020900 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:19.078979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.075179 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" podStartSLOduration=5.312841581 podStartE2EDuration="6.075160558s" totalImagesPullingTime="762.318977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.056605178 +0000 UTC m=+981.252296312" watchObservedRunningTime="2026-10-30 21:45:19.075160558 +0000 UTC m=+981.270851681" Oct 30 21:45:19.082975 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:19.080076 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:19.097927 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.095798 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.156974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.153257 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:45:19.156974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.154147 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.156974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.154484 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:45:19.156974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.155171 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.156974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.155506 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:45:19.209456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.181904 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerDied","Data":"9a19bdc0fe46f38ccf511ce6a1a736f24c114d2a9104b08eb7c3ca11ec95c9b5"} Oct 30 21:45:19.209456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.181998 6808 scope.go:122] "RemoveContainer" containerID="1852950ba60240370e31ccd48c5fa440bce52d7dc8f7d1f4def68b674ad0cdf3" Oct 30 21:45:19.209456 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:19.182838 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:19.209456 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:19.196328 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:19.230108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.228473 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg\" (UniqueName: \"kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg\") pod \"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd\" (UID: \"c03ce16f-58c0-4a0e-9daa-c80bd8856fdd\") " Oct 30 21:45:19.457062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.179829 6808 generic.go:356] "Generic (PLEG): container finished" podID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerID="9a19bdc0fe46f38ccf511ce6a1a736f24c114d2a9104b08eb7c3ca11ec95c9b5" exitCode=60 Oct 30 21:45:19.747511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.730911 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:45:19.747511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.731338 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:45:19.747511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.745919 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:45:19.757758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.757214 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerStarted","Data":"863904199bdafda23b983f26a95d7063b9d95e51b580795b22310363339cc1e5"} Oct 30 21:45:19.757758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.757739 6808 scope.go:122] "RemoveContainer" containerID="863904199bdafda23b983f26a95d7063b9d95e51b580795b22310363339cc1e5" Oct 30 21:45:19.769137 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:19.758190 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj_odh-ai-gateway-infra(a883830b-01ec-4c82-83c1-e4635dd47f18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" Oct 30 21:45:19.835905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.835045 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:19.845125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.844787 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.898087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.891251 6808 generic.go:356] "Generic (PLEG): container finished" podID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" containerID="3cfb6123eae4005df3778e729280426aea514c7cea75450c4df618d58a22f6bb" exitCode=137 Oct 30 21:45:19.898087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.892646 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:19.898087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:19.892786 6808 scope.go:122] "RemoveContainer" containerID="3cfb6123eae4005df3778e729280426aea514c7cea75450c4df618d58a22f6bb" Oct 30 21:45:19.915292 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:19.915256 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:20.005019 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:19.970265 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:20.119973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.112464 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.186296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.186234 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.211045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.207127 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:45:20.389733 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.389706 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:45:20.390013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.389994 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:45:20.485758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.485086 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.533175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.524118 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg" pod "c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" (UID: "c03ce16f-58c0-4a0e-9daa-c80bd8856fdd"). InnerVolumeSpecName "kube-api-access-pr2xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.605023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.598629 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2xg\" (UniqueName: \"kubernetes.io/projected/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd-kube-api-access-pr2xg\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.733087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.731726 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.733332 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.733231 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:45:20.749009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.734620 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:45:20.767288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.767169 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:45:20.804987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.804934 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:20.888051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.876048 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:45:20.968972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.968916 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj"] Oct 30 21:45:20.976209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.976150 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 21:45:20.986251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.986215 6808 generic.go:356] "Generic (PLEG): container finished" podID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerID="0e73653096a2ae7e8aab34374e9c4cd043fae270e589e58921f68935a35f350d" exitCode=60 Oct 30 21:45:20.986579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.986551 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerDied","Data":"0e73653096a2ae7e8aab34374e9c4cd043fae270e589e58921f68935a35f350d"} Oct 30 21:45:20.986764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.986747 6808 scope.go:122] "RemoveContainer" containerID="3e402fe34eb7a156086101a31e48cbcf6ab9e15f7f092f15a3caa4e96798181f" Oct 30 21:45:21.001106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:20.987414 6808 scope.go:122] "RemoveContainer" containerID="0e73653096a2ae7e8aab34374e9c4cd043fae270e589e58921f68935a35f350d" Oct 30 21:45:21.005924 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:21.005816 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-ss6qk_odh-ai-gateway-infra(b6f090b5-a6f1-4418-9db4-17c61ab26a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" Oct 30 21:45:21.009263 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.009216 6808 generic.go:356] "Generic (PLEG): container finished" podID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerID="0be3457b6949802cb1e1f7a32b0ae07fc507ca5036c6fd827661d2dd14e5cb3e" exitCode=60 Oct 30 21:45:21.013220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.009638 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerDied","Data":"0be3457b6949802cb1e1f7a32b0ae07fc507ca5036c6fd827661d2dd14e5cb3e"} Oct 30 21:45:21.013220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.011265 6808 scope.go:122] "RemoveContainer" containerID="0be3457b6949802cb1e1f7a32b0ae07fc507ca5036c6fd827661d2dd14e5cb3e" Oct 30 21:45:21.013220 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:21.011856 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b_odh-ai-gateway-infra(dac3094e-c37e-40e1-943b-f05e20dd6d07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" Oct 30 21:45:21.023444 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.023422 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" Oct 30 21:45:21.023716 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.023694 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml" event={"ID":"77252a6f-9b10-4481-b87a-4c2d9591a657","Type":"ContainerDied","Data":"a05bda5ea158df8b27da281f2688b69c217a229347b2979d56e66e025d447e0a"} Oct 30 21:45:21.042012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.024710 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca\") pod \"77252a6f-9b10-4481-b87a-4c2d9591a657\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " Oct 30 21:45:21.042409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.042364 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz\" (UniqueName: \"kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz\") pod \"77252a6f-9b10-4481-b87a-4c2d9591a657\" (UID: \"77252a6f-9b10-4481-b87a-4c2d9591a657\") " Oct 30 21:45:21.042844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.042806 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.044384 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.044305 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca" pod "77252a6f-9b10-4481-b87a-4c2d9591a657" (UID: "77252a6f-9b10-4481-b87a-4c2d9591a657"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.121386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.063087 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.139175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.136779 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.160876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.160840 6808 generic.go:356] "Generic (PLEG): container finished" podID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerID="863904199bdafda23b983f26a95d7063b9d95e51b580795b22310363339cc1e5" exitCode=60 Oct 30 21:45:21.176668 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.176632 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerDied","Data":"863904199bdafda23b983f26a95d7063b9d95e51b580795b22310363339cc1e5"} Oct 30 21:45:21.218553 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.218516 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:45:21.226176 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.226144 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.300855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.300810 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77252a6f-9b10-4481-b87a-4c2d9591a657-maas-api-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.522988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.522522 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 21:45:21.580993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.569730 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:45:21.580993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.573413 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.580993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.574258 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:45:21.610430 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.610403 6808 scope.go:122] "RemoveContainer" containerID="75d14c92ac7a41e15f90aeef34588f04c94f861fbb7c8f994bf89e3af6b0a266" Oct 30 21:45:21.615548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.615517 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz" pod "77252a6f-9b10-4481-b87a-4c2d9591a657" (UID: "77252a6f-9b10-4481-b87a-4c2d9591a657"). InnerVolumeSpecName "kube-api-access-xhzjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.764059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.752092 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:45:21.772235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.760193 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.772235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.760237 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:45:21.813806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.787915 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhzjz\" (UniqueName: \"kubernetes.io/projected/77252a6f-9b10-4481-b87a-4c2d9591a657-kube-api-access-xhzjz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.087275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.847033 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.087502 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:21.889150 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:22.096442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:21.900299 6808 scope.go:122] "RemoveContainer" containerID="9a19bdc0fe46f38ccf511ce6a1a736f24c114d2a9104b08eb7c3ca11ec95c9b5" Oct 30 21:45:22.222496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.222451 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.227084 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.227032 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.331604 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:22.265721 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-02f6e3\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" pod="opendatahub/test-curl-7342-02f6e3" Oct 30 21:45:22.677102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.675124 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 21:45:22.888971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.850578 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:45:22.888971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.860820 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.888971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.877914 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.927419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:22.927321 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.050060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.037309 6808 scope.go:122] "RemoveContainer" containerID="895e1916da548cc092eec9a9f9aa3fb28a8b3f6e0e1b3404c6c6f67429b36335" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.109722 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.110655 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03ce16f-58c0-4a0e-9daa-c80bd8856fdd" path="/var/lib/kubelet/pods/c03ce16f-58c0-4a0e-9daa-c80bd8856fdd/volumes" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:23.111400 6808 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.177s" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.111469 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.111567 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.111595 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.111634 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:45:23.120968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.120001 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk"] Oct 30 21:45:23.122372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.122351 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:45:23.131376 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.131264 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:45:23.136185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.134411 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:23.136185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.135997 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml"] Oct 30 21:45:23.148627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.143985 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7dfff0-29889945-jcpml"] Oct 30 21:45:23.148627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.147985 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b"] Oct 30 21:45:23.220018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.217892 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-137e4c"] Oct 30 21:45:23.220018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.218108 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7342-137e4c" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" containerName="test-curl-7342-137e4c" containerID="cri-o://b277566711c7e9b5995cb19e7a0d7d4925dbe04bf6a01f5e031a75889773e4fb" gracePeriod=2 Oct 30 21:45:23.229194 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:23.220634 6808 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77252a6f_9b10_4481_b87a_4c2d9591a657.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77252a6f_9b10_4481_b87a_4c2d9591a657.slice/crio-a05bda5ea158df8b27da281f2688b69c217a229347b2979d56e66e025d447e0a\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:23.229194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.224247 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-137e4c"] Oct 30 21:45:23.234184 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.234161 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca\") pod \"a883830b-01ec-4c82-83c1-e4635dd47f18\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " Oct 30 21:45:23.234494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.234477 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn\" (UniqueName: \"kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn\") pod \"a883830b-01ec-4c82-83c1-e4635dd47f18\" (UID: \"a883830b-01ec-4c82-83c1-e4635dd47f18\") " Oct 30 21:45:23.236432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.236408 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:45:23.274971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.268219 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:45:23.298505 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.298473 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca" pod "a883830b-01ec-4c82-83c1-e4635dd47f18" (UID: "a883830b-01ec-4c82-83c1-e4635dd47f18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:23.328599 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.328538 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn" pod "a883830b-01ec-4c82-83c1-e4635dd47f18" (UID: "a883830b-01ec-4c82-83c1-e4635dd47f18"). InnerVolumeSpecName "kube-api-access-bl8sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:23.344830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.340259 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a883830b-01ec-4c82-83c1-e4635dd47f18-maas-api-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.344830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.340288 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl8sn\" (UniqueName: \"kubernetes.io/projected/a883830b-01ec-4c82-83c1-e4635dd47f18-kube-api-access-bl8sn\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.430985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.430942 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-d40db9"] Oct 30 21:45:23.432292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.432271 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:23.432442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.432429 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:23.432740 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.432728 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" containerName="test-curl-7342-137e4c" Oct 30 21:45:23.432841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.432830 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" containerName="test-curl-7342-137e4c" Oct 30 21:45:23.432959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.432937 6808 cpu_manager.go:414] "containerMap: removing container" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:23.433043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.433032 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:23.433141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.433133 6808 cpu_manager.go:414] "containerMap: removing container" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:23.433206 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.433198 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:23.433294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.433286 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:23.433363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.433354 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:23.434532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.434515 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:23.435036 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.435020 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:23.435167 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.435154 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:23.435270 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.435258 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:23.435363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.435351 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" containerName="test-curl-7342-137e4c" Oct 30 21:45:23.516549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.516468 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-d40db9"] Oct 30 21:45:23.516773 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:23.516481 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-137e4c\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:23.517917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.516578 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:23.619201 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:23.614727 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-137e4c\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:23.719701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.719659 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjzd\" (UniqueName: \"kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd\") pod \"test-curl-7342-d40db9\" (UID: \"7719c057-5530-49b6-bbab-c9bc706cde8f\") " pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:23.984187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:23.981124 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjzd\" (UniqueName: \"kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd\") pod \"test-curl-7342-d40db9\" (UID: \"7719c057-5530-49b6-bbab-c9bc706cde8f\") " pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:24.075506 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.075109 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjzd\" (UniqueName: \"kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd\") pod \"test-curl-7342-d40db9\" (UID: \"7719c057-5530-49b6-bbab-c9bc706cde8f\") " pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:24.121023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.117796 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" path="/var/lib/kubelet/pods/77252a6f-9b10-4481-b87a-4c2d9591a657/volumes" Oct 30 21:45:24.187149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.187106 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:24.318328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.318297 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:24.332379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.332354 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" Oct 30 21:45:24.332556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.332534 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk" event={"ID":"b6f090b5-a6f1-4418-9db4-17c61ab26a73","Type":"ContainerDied","Data":"c7890085d145bf9c0e457d0aac2a64e8f86e543df04de16b0a9078d9a2b21183"} Oct 30 21:45:24.332616 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.332589 6808 scope.go:122] "RemoveContainer" containerID="0e73653096a2ae7e8aab34374e9c4cd043fae270e589e58921f68935a35f350d" Oct 30 21:45:24.334287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.334265 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:24.403987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.396545 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" event={"ID":"dac3094e-c37e-40e1-943b-f05e20dd6d07","Type":"ContainerDied","Data":"2d563447915d93969c64604fe77068701b48f3b10cb8c99cdc21f4361196f394"} Oct 30 21:45:24.403987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.396653 6808 scope.go:122] "RemoveContainer" containerID="0be3457b6949802cb1e1f7a32b0ae07fc507ca5036c6fd827661d2dd14e5cb3e" Oct 30 21:45:24.424372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.424300 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca\") pod \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " Oct 30 21:45:24.427142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.424689 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f\" (UniqueName: \"kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f\") pod \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\" (UID: \"b6f090b5-a6f1-4418-9db4-17c61ab26a73\") " Oct 30 21:45:24.436440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.436277 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" Oct 30 21:45:24.482012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.437290 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj" event={"ID":"a883830b-01ec-4c82-83c1-e4635dd47f18","Type":"ContainerDied","Data":"2f055ec8d34666bcd6c43b8089e53431228ee4d1b5cd3033f10b40229f405ebe"} Oct 30 21:45:24.482466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.482436 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca" pod "b6f090b5-a6f1-4418-9db4-17c61ab26a73" (UID: "b6f090b5-a6f1-4418-9db4-17c61ab26a73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:24.491394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.488685 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f090b5-a6f1-4418-9db4-17c61ab26a73-maas-api-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.510125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.509028 6808 scope.go:122] "RemoveContainer" containerID="863904199bdafda23b983f26a95d7063b9d95e51b580795b22310363339cc1e5" Oct 30 21:45:24.585042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.577117 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f" pod "b6f090b5-a6f1-4418-9db4-17c61ab26a73" (UID: "b6f090b5-a6f1-4418-9db4-17c61ab26a73"). InnerVolumeSpecName "kube-api-access-wqw8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:24.597838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.597806 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca\") pod \"dac3094e-c37e-40e1-943b-f05e20dd6d07\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " Oct 30 21:45:24.598204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.598183 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq\" (UniqueName: \"kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq\") pod \"dac3094e-c37e-40e1-943b-f05e20dd6d07\" (UID: \"dac3094e-c37e-40e1-943b-f05e20dd6d07\") " Oct 30 21:45:24.606259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.599355 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqw8f\" (UniqueName: \"kubernetes.io/projected/b6f090b5-a6f1-4418-9db4-17c61ab26a73-kube-api-access-wqw8f\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.606416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.600707 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq" pod "dac3094e-c37e-40e1-943b-f05e20dd6d07" (UID: "dac3094e-c37e-40e1-943b-f05e20dd6d07"). InnerVolumeSpecName "kube-api-access-6hmhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:24.606518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.601203 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca" pod "dac3094e-c37e-40e1-943b-f05e20dd6d07" (UID: "dac3094e-c37e-40e1-943b-f05e20dd6d07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:24.623741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.623713 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj"] Oct 30 21:45:24.639243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.639211 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbf428-29889945-z58mj"] Oct 30 21:45:24.693461 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.693432 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk"] Oct 30 21:45:24.704726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.704694 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ss6qk"] Oct 30 21:45:24.708773 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.708719 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac3094e-c37e-40e1-943b-f05e20dd6d07-maas-api-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.708773 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.708747 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmhq\" (UniqueName: \"kubernetes.io/projected/dac3094e-c37e-40e1-943b-f05e20dd6d07-kube-api-access-6hmhq\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.812028 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:45:24.811991 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7719c057_5530_49b6_bbab_c9bc706cde8f.slice/crio-2c879a640790aabb2eeab042289392c906629b9ac80c471ff3eb75c3d33b7fe5 WatchSource:0}: Error finding container 2c879a640790aabb2eeab042289392c906629b9ac80c471ff3eb75c3d33b7fe5: Status 404 returned error can't find the container with id 2c879a640790aabb2eeab042289392c906629b9ac80c471ff3eb75c3d33b7fe5 Oct 30 21:45:24.813350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:24.813317 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-d40db9"] Oct 30 21:45:25.483024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.478996 6808 generic.go:356] "Generic (PLEG): container finished" podID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" containerID="b277566711c7e9b5995cb19e7a0d7d4925dbe04bf6a01f5e031a75889773e4fb" exitCode=137 Oct 30 21:45:25.483024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.482727 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b" Oct 30 21:45:25.493441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.493412 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-d40db9" event={"ID":"7719c057-5530-49b6-bbab-c9bc706cde8f","Type":"ContainerStarted","Data":"34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8"} Oct 30 21:45:25.493583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.493455 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-d40db9" event={"ID":"7719c057-5530-49b6-bbab-c9bc706cde8f","Type":"ContainerStarted","Data":"2c879a640790aabb2eeab042289392c906629b9ac80c471ff3eb75c3d33b7fe5"} Oct 30 21:45:25.513533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.513498 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b"] Oct 30 21:45:25.518764 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.518739 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-dc3edf-29889945-h948b"] Oct 30 21:45:25.537143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.537072 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7342-d40db9" podStartSLOduration=2.537055489 podStartE2EDuration="2.537055489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:25.531673655 +0000 UTC m=+987.727364769" watchObservedRunningTime="2026-10-30 21:45:25.537055489 +0000 UTC m=+987.732746607" Oct 30 21:45:25.610532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.610497 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:25.783020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.780939 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8\" (UniqueName: \"kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8\") pod \"bf1e6c75-adcb-42d3-a378-f3b113eb0e42\" (UID: \"bf1e6c75-adcb-42d3-a378-f3b113eb0e42\") " Oct 30 21:45:25.784689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.784646 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8" pod "bf1e6c75-adcb-42d3-a378-f3b113eb0e42" (UID: "bf1e6c75-adcb-42d3-a378-f3b113eb0e42"). InnerVolumeSpecName "kube-api-access-ksnj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:25.914969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:25.888886 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksnj8\" (UniqueName: \"kubernetes.io/projected/bf1e6c75-adcb-42d3-a378-f3b113eb0e42-kube-api-access-ksnj8\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:26.035475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.035378 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" path="/var/lib/kubelet/pods/a883830b-01ec-4c82-83c1-e4635dd47f18/volumes" Oct 30 21:45:26.044321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.044255 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" path="/var/lib/kubelet/pods/b6f090b5-a6f1-4418-9db4-17c61ab26a73/volumes" Oct 30 21:45:26.045600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.045575 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1e6c75-adcb-42d3-a378-f3b113eb0e42" path="/var/lib/kubelet/pods/bf1e6c75-adcb-42d3-a378-f3b113eb0e42/volumes" Oct 30 21:45:26.046484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.046464 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" path="/var/lib/kubelet/pods/dac3094e-c37e-40e1-943b-f05e20dd6d07/volumes" Oct 30 21:45:26.503664 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.503635 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-137e4c" Oct 30 21:45:26.504179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.503684 6808 scope.go:122] "RemoveContainer" containerID="b277566711c7e9b5995cb19e7a0d7d4925dbe04bf6a01f5e031a75889773e4fb" Oct 30 21:45:26.517550 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.517523 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-d40db9"] Oct 30 21:45:26.529085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.529056 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-d40db9"] Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.703046 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-61570d"] Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704206 6808 cpu_manager.go:414] "containerMap: removing container" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704229 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704252 6808 cpu_manager.go:414] "containerMap: removing container" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704259 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704270 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704278 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704301 6808 cpu_manager.go:414] "containerMap: removing container" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704308 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704327 6808 cpu_manager.go:414] "containerMap: removing container" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704334 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704358 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704365 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704382 6808 cpu_manager.go:414] "containerMap: removing container" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" containerName="test-curl-7342-d40db9" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704389 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" containerName="test-curl-7342-d40db9" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704406 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704413 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704424 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704432 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704607 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704625 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="77252a6f-9b10-4481-b87a-4c2d9591a657" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704638 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" containerName="test-curl-7342-d40db9" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704651 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704661 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704670 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a883830b-01ec-4c82-83c1-e4635dd47f18" containerName="cleanup" Oct 30 21:45:26.711973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.704687 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:26.732197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.717311 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-61570d"] Oct 30 21:45:26.732197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.717425 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:26.840738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.840616 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7d7\" (UniqueName: \"kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7\") pod \"test-curl-7342-61570d\" (UID: \"245dbc0f-82f5-463f-b032-9fce38526e5e\") " pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:26.942708 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.942672 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7d7\" (UniqueName: \"kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7\") pod \"test-curl-7342-61570d\" (UID: \"245dbc0f-82f5-463f-b032-9fce38526e5e\") " pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:26.959208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:26.959173 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7d7\" (UniqueName: \"kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7\") pod \"test-curl-7342-61570d\" (UID: \"245dbc0f-82f5-463f-b032-9fce38526e5e\") " pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:27.066824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:27.066778 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:27.359708 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:27.359676 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-61570d"] Oct 30 21:45:27.360515 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:45:27.360486 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod245dbc0f_82f5_463f_b032_9fce38526e5e.slice/crio-d17ef87477ffa41805c356e0b0b74abe3d98acb686c0fdbbce9ddeed6efbad19 WatchSource:0}: Error finding container d17ef87477ffa41805c356e0b0b74abe3d98acb686c0fdbbce9ddeed6efbad19: Status 404 returned error can't find the container with id d17ef87477ffa41805c356e0b0b74abe3d98acb686c0fdbbce9ddeed6efbad19 Oct 30 21:45:27.528533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:27.527200 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-61570d" event={"ID":"245dbc0f-82f5-463f-b032-9fce38526e5e","Type":"ContainerStarted","Data":"0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17"} Oct 30 21:45:27.528533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:27.527256 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-61570d" event={"ID":"245dbc0f-82f5-463f-b032-9fce38526e5e","Type":"ContainerStarted","Data":"d17ef87477ffa41805c356e0b0b74abe3d98acb686c0fdbbce9ddeed6efbad19"} Oct 30 21:45:27.531280 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:27.530902 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7342-d40db9" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" containerName="test-curl-7342-d40db9" containerID="cri-o://34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8" gracePeriod=2 Oct 30 21:45:27.586316 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:27.577172 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-d40db9\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:27.927973 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:27.927919 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-d40db9\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:28.334029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:28.330629 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7342-61570d" podStartSLOduration=2.330606957 podStartE2EDuration="2.330606957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:27.554128188 +0000 UTC m=+989.749819308" watchObservedRunningTime="2026-10-30 21:45:28.330606957 +0000 UTC m=+990.526298072" Oct 30 21:45:28.337972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:28.336824 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-61570d"] Oct 30 21:45:28.345496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:28.344497 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-61570d"] Oct 30 21:45:29.566869 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:29.566801 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7342-61570d" podUID="245dbc0f-82f5-463f-b032-9fce38526e5e" containerName="test-curl-7342-61570d" containerID="cri-o://0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17" gracePeriod=2 Oct 30 21:45:29.592234 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:29.592196 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-61570d\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="245dbc0f-82f5-463f-b032-9fce38526e5e" pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:29.946864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:29.946832 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:30.033658 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.033624 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd\" (UniqueName: \"kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd\") pod \"7719c057-5530-49b6-bbab-c9bc706cde8f\" (UID: \"7719c057-5530-49b6-bbab-c9bc706cde8f\") " Oct 30 21:45:30.036598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.036530 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd" pod "7719c057-5530-49b6-bbab-c9bc706cde8f" (UID: "7719c057-5530-49b6-bbab-c9bc706cde8f"). InnerVolumeSpecName "kube-api-access-rbjzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:30.136775 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.136745 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjzd\" (UniqueName: \"kubernetes.io/projected/7719c057-5530-49b6-bbab-c9bc706cde8f-kube-api-access-rbjzd\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:30.576824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.576728 6808 generic.go:356] "Generic (PLEG): container finished" podID="7719c057-5530-49b6-bbab-c9bc706cde8f" containerID="34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8" exitCode=137 Oct 30 21:45:30.576824 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.576776 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:30.577416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.576995 6808 scope.go:122] "RemoveContainer" containerID="34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8" Oct 30 21:45:30.589555 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:30.589526 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-d40db9\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:30.592042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.592024 6808 scope.go:122] "RemoveContainer" containerID="34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8" Oct 30 21:45:30.592354 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:30.592336 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8\": container with ID starting with 34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8 not found: ID does not exist" containerID="34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8" Oct 30 21:45:30.592400 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:30.592361 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8"} err="failed to get container status \"34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8\": rpc error: code = NotFound desc = could not find container \"34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8\": container with ID starting with 34fe73357915c04effd69f69e19b20cfb88babdaf09037ce30809bb5323089a8 not found: ID does not exist" Oct 30 21:45:31.588516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.588477 6808 generic.go:356] "Generic (PLEG): container finished" podID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerID="c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1" exitCode=0 Oct 30 21:45:31.588938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.588548 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" event={"ID":"cd168855-dfa6-412d-a18b-03a29a6bb7ca","Type":"ContainerDied","Data":"c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1"} Oct 30 21:45:31.589026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.589015 6808 scope.go:122] "RemoveContainer" containerID="c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1" Oct 30 21:45:31.590639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.590608 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:31.590762 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.590691 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:31.590762 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.590740 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:45:31.591482 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:31.591459 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-d40db9\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:31.899725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.899693 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:31.902291 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:31.902252 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7342-d40db9\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" pod="opendatahub/test-curl-7342-d40db9" Oct 30 21:45:31.931574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.931542 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7719c057-5530-49b6-bbab-c9bc706cde8f" path="/var/lib/kubelet/pods/7719c057-5530-49b6-bbab-c9bc706cde8f/volumes" Oct 30 21:45:31.996977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:31.996046 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7\" (UniqueName: \"kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7\") pod \"245dbc0f-82f5-463f-b032-9fce38526e5e\" (UID: \"245dbc0f-82f5-463f-b032-9fce38526e5e\") " Oct 30 21:45:32.018001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.006582 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7" pod "245dbc0f-82f5-463f-b032-9fce38526e5e" (UID: "245dbc0f-82f5-463f-b032-9fce38526e5e"). InnerVolumeSpecName "kube-api-access-4m7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:32.100923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.100819 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7d7\" (UniqueName: \"kubernetes.io/projected/245dbc0f-82f5-463f-b032-9fce38526e5e-kube-api-access-4m7d7\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:32.596968 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.596909 6808 generic.go:356] "Generic (PLEG): container finished" podID="245dbc0f-82f5-463f-b032-9fce38526e5e" containerID="0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17" exitCode=137 Oct 30 21:45:32.597435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.596982 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-61570d" Oct 30 21:45:32.597435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.597019 6808 scope.go:122] "RemoveContainer" containerID="0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17" Oct 30 21:45:32.599461 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.599425 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" event={"ID":"cd168855-dfa6-412d-a18b-03a29a6bb7ca","Type":"ContainerStarted","Data":"1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a"} Oct 30 21:45:32.608447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.608431 6808 scope.go:122] "RemoveContainer" containerID="0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17" Oct 30 21:45:32.608714 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:32.608697 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17\": container with ID starting with 0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17 not found: ID does not exist" containerID="0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17" Oct 30 21:45:32.608802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:32.608723 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17"} err="failed to get container status \"0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17\": rpc error: code = NotFound desc = could not find container \"0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17\": container with ID starting with 0dbf1c16e88427cd60e883df0f579c0b3ad1b386d902c733339394ac16fc8c17 not found: ID does not exist" Oct 30 21:45:33.152971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:33.149016 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:45:33.174974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:33.170731 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:45:33.174974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:33.170807 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 21:45:33.939233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:33.939203 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245dbc0f-82f5-463f-b032-9fce38526e5e" path="/var/lib/kubelet/pods/245dbc0f-82f5-463f-b032-9fce38526e5e/volumes" Oct 30 21:45:34.158971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:34.157045 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:45:34.158971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:34.157105 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 21:45:35.164974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:35.163350 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:45:35.164974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:35.163432 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 21:45:35.704468 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:45:35.704430 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:45:36.245972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.239590 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:45:36.245972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.239677 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.710202 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w"] Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.712181 6808 cpu_manager.go:414] "containerMap: removing container" podUID="245dbc0f-82f5-463f-b032-9fce38526e5e" containerName="test-curl-7342-61570d" Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.712211 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245dbc0f-82f5-463f-b032-9fce38526e5e" containerName="test-curl-7342-61570d" Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.712768 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="245dbc0f-82f5-463f-b032-9fce38526e5e" containerName="test-curl-7342-61570d" Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.712800 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac3094e-c37e-40e1-943b-f05e20dd6d07" containerName="cleanup" Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.712828 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f090b5-a6f1-4418-9db4-17c61ab26a73" containerName="cleanup" Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.736385 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w"] Oct 30 21:45:36.738032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.736523 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.759432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.759400 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-08aafeac\"/\"kube-root-ca.crt\"" Oct 30 21:45:36.764796 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.762443 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-08aafeac\"/\"e2e-lifecycle-model-08aafeac-kserve-self-signed-certs\"" Oct 30 21:45:36.771051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.763561 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-08aafeac\"/\"default-dockercfg-gxwg4\"" Oct 30 21:45:36.771304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.763642 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-08aafeac\"/\"openshift-service-ca.crt\"" Oct 30 21:45:36.825197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.825157 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.825560 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.825539 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.825829 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.825812 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.826044 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.826028 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.826190 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.826172 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.826298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.826282 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2mq\" (UniqueName: \"kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.929457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.929422 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.929808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.929783 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.930458 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.930285 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.930911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.930849 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.931142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.931124 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.931282 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.931266 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2mq\" (UniqueName: \"kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.954861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.931897 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.958059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.958031 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.958819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.958794 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.985537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.957806 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:36.985751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:36.985457 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:37.002045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.001922 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2mq\" (UniqueName: \"kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq\") pod \"e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:37.138840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.138807 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:37.149407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.149368 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:45:37.149612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.149588 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 21:45:37.555054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.538994 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w"] Oct 30 21:45:37.556690 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:45:37.556647 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c551711_d8a4_4824_aee5_d958286e738d.slice/crio-15a6408b2435018ab1b65ac949890b903feee6c8af4905215b3f4b715bf3ef67 WatchSource:0}: Error finding container 15a6408b2435018ab1b65ac949890b903feee6c8af4905215b3f4b715bf3ef67: Status 404 returned error can't find the container with id 15a6408b2435018ab1b65ac949890b903feee6c8af4905215b3f4b715bf3ef67 Oct 30 21:45:37.823053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.822931 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" event={"ID":"3c551711-d8a4-4824-aee5-d958286e738d","Type":"ContainerStarted","Data":"a353c91f81eb6ba23c994573bad78fa51e9c238b720b98085d6c7bc3b60704d5"} Oct 30 21:45:37.823053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:37.823006 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" event={"ID":"3c551711-d8a4-4824-aee5-d958286e738d","Type":"ContainerStarted","Data":"15a6408b2435018ab1b65ac949890b903feee6c8af4905215b3f4b715bf3ef67"} Oct 30 21:45:38.152186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:38.152155 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:38.152757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:38.152731 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:45:38.152996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:38.152971 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 21:45:38.385442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:38.383089 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:45:39.155104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:39.155072 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:45:39.156480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:39.156458 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:45:39.185342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:39.185291 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podStartSLOduration=3.185277003 podStartE2EDuration="3.185277003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:37.858167551 +0000 UTC m=+1000.053858671" watchObservedRunningTime="2026-10-30 21:45:39.185277003 +0000 UTC m=+1001.380968177" Oct 30 21:45:47.139079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:47.138998 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:47.153345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:47.153315 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:45:47.982572 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:45:47.982543 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:46:00.776973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:00.774072 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:00.928980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:00.928081 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.033018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.031476 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.085617 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.085570 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.135439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.135399 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:46:01.155973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.152683 6808 scope.go:122] "RemoveContainer" containerID="4d4f33e924ad4ec4ba4753175268c36b20b0d5ddc616c4840be705761e659d3d" Oct 30 21:46:01.195620 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.194143 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.229203 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.229119 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.537181 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.535101 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.787198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.754743 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:01.787198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.754847 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.787198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.785800 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:46:01.787198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.786056 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:46:01.857995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.838015 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.903207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.887217 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.936447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.907638 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:01.936447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:01.930023 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.155973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.136017 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.155973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.144584 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.211999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.206120 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:46:02.243660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.242353 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:46:02.593979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.578448 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:46:02.649027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.636925 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:46:02.660137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.658890 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:46:02.679998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.677733 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:46:02.688090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.682234 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:46:02.688316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.688296 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:46:02.807415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.805681 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:46:02.903631 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.903580 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.915152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:02.915104 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:46:03.010465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.004163 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:03.281115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.255480 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:46:03.281115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.259021 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:46:03.285825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.285783 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:46:03.581140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.581098 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:46:03.581624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.581606 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:46:03.619382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.618976 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:46:03.628488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.628456 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:46:03.628819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.628518 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:46:03.633447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.633421 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:46:03.650057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:03.636578 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:46:05.715871 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:46:05.715774 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:46:11.636973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:11.631338 6808 generic.go:356] "Generic (PLEG): container finished" podID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerID="25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7" exitCode=0 Oct 30 21:46:11.636973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:11.631544 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" event={"ID":"4177f737-6bc7-4647-b1d8-351e85c33cee","Type":"ContainerDied","Data":"25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7"} Oct 30 21:46:11.636973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:11.632091 6808 scope.go:122] "RemoveContainer" containerID="25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7" Oct 30 21:46:11.664972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:11.658667 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:11.664972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:11.658775 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:11.664972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:11.658849 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:46:12.647917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:12.646395 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:12.749350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:12.749310 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" event={"ID":"4177f737-6bc7-4647-b1d8-351e85c33cee","Type":"ContainerStarted","Data":"3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6"} Oct 30 21:46:12.984428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:12.984309 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.029832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:13.029710 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:13.377128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:13.377043 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:46:13.377503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:13.377385 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.377718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:13.377693 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:14.376397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:14.376351 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.377096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:14.377059 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:15.375491 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:15.375451 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.375836 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:15.375775 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:16.087783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:16.087710 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:46:16.374791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:16.374704 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:16.375070 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:16.375036 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:17.375516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.375475 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:17.376139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.376109 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:17.902671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.902640 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg"] Oct 30 21:46:17.906286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.906264 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:17.918717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.918693 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg"] Oct 30 21:46:17.977035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.976958 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6cf5f0-dockercfg-wtstc\"" Oct 30 21:46:17.984572 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:17.978135 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6cf5f0\"" Oct 30 21:46:18.013974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.006142 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqvlj\" (UniqueName: \"kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj\") pod \"payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:18.013974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.006444 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:18.133974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.132806 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvlj\" (UniqueName: \"kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj\") pod \"payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:18.133974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.133201 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:18.134241 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.134221 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:18.408041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.406077 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:18.408041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.406157 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:18.471973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.456940 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqvlj\" (UniqueName: \"kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj\") pod \"payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:18.541982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:18.536242 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:19.417285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.417246 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg"] Oct 30 21:46:19.420150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.420118 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:19.420391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.420351 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:19.899142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.899104 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" event={"ID":"14e04609-0dd9-47e2-a05c-2d5762433b29","Type":"ContainerStarted","Data":"9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8"} Oct 30 21:46:19.899343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.899173 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:19.899343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.899196 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" event={"ID":"14e04609-0dd9-47e2-a05c-2d5762433b29","Type":"ContainerStarted","Data":"11e180f8b5f24f6397db21ffbe319465f467336dd66e926d48da4884757cf042"} Oct 30 21:46:19.929294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:19.929234 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" podStartSLOduration=2.929215298 podStartE2EDuration="2.929215298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:19.922338048 +0000 UTC m=+1042.118029159" watchObservedRunningTime="2026-10-30 21:46:19.929215298 +0000 UTC m=+1042.124906418" Oct 30 21:46:20.373841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:20.373760 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:46:20.373841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:20.373822 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 21:46:21.378717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:21.378688 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:46:21.379778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:21.379757 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:46:28.569575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.569543 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs"] Oct 30 21:46:28.574043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.574019 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.590586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.590554 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-6cf5f0-serving-cert\"" Oct 30 21:46:28.595066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.595040 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs"] Oct 30 21:46:28.633035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.632993 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2z2\" (UniqueName: \"kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.633328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.633294 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.633498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.633478 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.633815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.633783 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.738346 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.738305 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2z2\" (UniqueName: \"kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.738722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.738693 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.738861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.738740 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.739002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.738918 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.739862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.739839 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.740458 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:46:28.740411 6808 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-serving-cert: secret "maas-api-e2e-shared-a-w4-6cf5f0-serving-cert" not found Oct 30 21:46:28.740576 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:46:28.740484 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls podName:74161860-6929-41a8-92e7-69d171cf972b nodeName:}" failed. No retries permitted until 2026-10-30 21:46:29.240464284 +0000 UTC m=+1051.436155395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls") pod "maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" (UID: "74161860-6929-41a8-92e7-69d171cf972b") : secret "maas-api-e2e-shared-a-w4-6cf5f0-serving-cert" not found Oct 30 21:46:28.790752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.790716 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:28.804973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:28.791657 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2z2\" (UniqueName: \"kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:29.274974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:29.266521 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:29.274974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:29.272760 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:29.543026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:29.538595 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:30.162025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:30.161998 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs"] Oct 30 21:46:30.162394 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:46:30.161995 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74161860_6929_41a8_92e7_69d171cf972b.slice/crio-206c4334a47df7ba2ad74a89633333f38f38793db5081a6ae03f288625e3ce21 WatchSource:0}: Error finding container 206c4334a47df7ba2ad74a89633333f38f38793db5081a6ae03f288625e3ce21: Status 404 returned error can't find the container with id 206c4334a47df7ba2ad74a89633333f38f38793db5081a6ae03f288625e3ce21 Oct 30 21:46:30.910066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:30.910041 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:46:31.150989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:31.150937 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" event={"ID":"74161860-6929-41a8-92e7-69d171cf972b","Type":"ContainerStarted","Data":"5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0"} Oct 30 21:46:31.151249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:31.151231 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" event={"ID":"74161860-6929-41a8-92e7-69d171cf972b","Type":"ContainerStarted","Data":"206c4334a47df7ba2ad74a89633333f38f38793db5081a6ae03f288625e3ce21"} Oct 30 21:46:31.151641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:31.151615 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:31.175165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:31.175110 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" podStartSLOduration=2.634443435 podStartE2EDuration="3.175092375s" totalImagesPullingTime="540.64894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:31.17399436 +0000 UTC m=+1053.369685482" watchObservedRunningTime="2026-10-30 21:46:31.175092375 +0000 UTC m=+1053.370783495" Oct 30 21:46:35.701520 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:46:35.701481 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:46:37.169530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:37.169493 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:46:53.324514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.324480 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf"] Oct 30 21:46:53.329385 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.329360 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.334377 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.334350 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-aad4dd-dockercfg-mbfwc\"" Oct 30 21:46:53.334680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.334653 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-aad4dd\"" Oct 30 21:46:53.337533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.337511 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf"] Oct 30 21:46:53.576032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.575566 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.576032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.575754 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fdz\" (UniqueName: \"kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz\") pod \"payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.729172 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.729109 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.738291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.738246 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fdz\" (UniqueName: \"kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz\") pod \"payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.753988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.734803 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.887017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.886660 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fdz\" (UniqueName: \"kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz\") pod \"payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:53.947184 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:53.947154 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:54.251030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:54.250941 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf"] Oct 30 21:46:54.256343 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:46:54.256308 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd6ff63_fbfe_4ad7_9a5e_f667aadd88f3.slice/crio-e55148abf029b5d7147f27c183932fd0901d63fb1ae750bf984e6be6cbc99b74 WatchSource:0}: Error finding container e55148abf029b5d7147f27c183932fd0901d63fb1ae750bf984e6be6cbc99b74: Status 404 returned error can't find the container with id e55148abf029b5d7147f27c183932fd0901d63fb1ae750bf984e6be6cbc99b74 Oct 30 21:46:54.792247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:54.792216 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" event={"ID":"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3","Type":"ContainerStarted","Data":"1906f750cb8b19c59380a0f3400e4af23de446b663d0ac8452ca9450fe002e2e"} Oct 30 21:46:54.792671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:54.792268 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" event={"ID":"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3","Type":"ContainerStarted","Data":"e55148abf029b5d7147f27c183932fd0901d63fb1ae750bf984e6be6cbc99b74"} Oct 30 21:46:54.792671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:54.792314 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:46:54.815873 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:46:54.815823 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" podStartSLOduration=1.815809161 podStartE2EDuration="1.815809161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.813378232 +0000 UTC m=+1077.009069352" watchObservedRunningTime="2026-10-30 21:46:54.815809161 +0000 UTC m=+1077.011500280" Oct 30 21:47:05.821375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:05.819638 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:47:05.859377 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:47:05.859336 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:47:08.907711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:08.907660 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq"] Oct 30 21:47:08.932270 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:08.932217 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq"] Oct 30 21:47:08.932607 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:08.932580 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:08.948415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:08.948385 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd3c4c-openshift-default-dockercfg-xn6l7\"" Oct 30 21:47:09.059974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.053875 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.059974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.054050 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.078421 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.078716 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.079052 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vhw\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.079120 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.079303 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.079348 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.079977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.079401 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.180886 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.180941 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181076 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181175 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181204 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181437 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181280 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181437 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181331 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181466 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vhw\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.181544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.181511 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.225987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.212563 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.239037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.213182 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.239253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.213404 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.239348 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.214044 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.239436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.214713 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.239520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.214969 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.239601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.223086 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.250589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.250558 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.295734 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.295697 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vhw\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw\") pod \"e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:09.588344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:09.588248 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:10.290109 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:47:10.290076 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46ac04d_5ae1_43f1_be29_5a08872cb890.slice/crio-1bb237dc55bcd27d36eac21caf5b918466f7d8c06121f1bd1d89218a55aeed2c WatchSource:0}: Error finding container 1bb237dc55bcd27d36eac21caf5b918466f7d8c06121f1bd1d89218a55aeed2c: Status 404 returned error can't find the container with id 1bb237dc55bcd27d36eac21caf5b918466f7d8c06121f1bd1d89218a55aeed2c Oct 30 21:47:10.290989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:10.290944 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq"] Oct 30 21:47:10.293644 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:10.293605 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:10.293744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:10.293694 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:10.293744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:10.293736 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:11.288984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.288811 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" event={"ID":"f46ac04d-5ae1-43f1-be29-5a08872cb890","Type":"ContainerStarted","Data":"8c370170f2eb312ffca0ef4482d9fa8342e81e5965cc3cf17c56935b8ead6714"} Oct 30 21:47:11.288984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.288878 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" event={"ID":"f46ac04d-5ae1-43f1-be29-5a08872cb890","Type":"ContainerStarted","Data":"1bb237dc55bcd27d36eac21caf5b918466f7d8c06121f1bd1d89218a55aeed2c"} Oct 30 21:47:11.571027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.567299 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podStartSLOduration=3.567268612 podStartE2EDuration="3.567268612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:11.325161548 +0000 UTC m=+1093.520852668" watchObservedRunningTime="2026-10-30 21:47:11.567268612 +0000 UTC m=+1093.762959732" Oct 30 21:47:11.583118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.583079 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs"] Oct 30 21:47:11.600256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.600228 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.609166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.608981 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs"] Oct 30 21:47:11.670806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.670776 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:11.680698 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.680666 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:11.701935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.701900 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7386906d-openshift-default-dockercfg-ffr97\"" Oct 30 21:47:11.784469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.784434 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgsg\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.784670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.784613 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.784670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.784655 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.784795 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.784699 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.784877 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.784860 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.785252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.785223 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.785381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.785348 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.785453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.785405 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.785516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.785465 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.886831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.886799 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.886899 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887000 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887045 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887077 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgsg\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887128 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887162 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887210 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887413 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.887928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887903 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.888131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.887982 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.888288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.888269 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.888394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.888372 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.888737 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.888693 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.890116 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.890085 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.890308 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.890288 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.901003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.900976 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgsg\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.901146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.901126 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:11.976704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:11.976666 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:12.326028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:12.325323 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:12.332983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:12.326862 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:47:12.365970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:12.363907 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs"] Oct 30 21:47:12.386843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:12.386803 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:12.387051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:12.386899 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:12.387051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:12.387002 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:13.526381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:13.526308 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" event={"ID":"0c01e6b8-5944-4e27-8328-e240f99889cc","Type":"ContainerStarted","Data":"84075b4f6cdf0bb7039c12ae5b51bce0f07eddc30119a35501c59d9635b9d985"} Oct 30 21:47:13.526888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:13.526414 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" event={"ID":"0c01e6b8-5944-4e27-8328-e240f99889cc","Type":"ContainerStarted","Data":"b8c2257cb3d33f09e98495c3b7e0eb13659e56f4d2437b20ca6e9ded9af14148"} Oct 30 21:47:13.979210 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:13.979172 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:13.979762 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:13.979732 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:47:13.979890 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:13.979794 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:47:14.978244 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:14.978211 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:47:14.978747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:14.978265 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:47:15.175981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.172286 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podStartSLOduration=4.172265433 podStartE2EDuration="4.172265433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.577288222 +0000 UTC m=+1095.772979342" watchObservedRunningTime="2026-10-30 21:47:15.172265433 +0000 UTC m=+1097.367956553" Oct 30 21:47:15.178363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.177481 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj"] Oct 30 21:47:15.186216 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.186177 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj"] Oct 30 21:47:15.186515 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.186497 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.198645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.198159 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd3c4c\"" Oct 30 21:47:15.206717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.205386 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.206717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.205485 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspjz\" (UniqueName: \"kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz\") pod \"payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.330054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.312170 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.330054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.312282 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspjz\" (UniqueName: \"kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz\") pod \"payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.330054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.314034 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.487972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.472646 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspjz\" (UniqueName: \"kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz\") pod \"payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.557974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.556434 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:15.995819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.995360 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:47:15.995819 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:15.995423 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:47:16.378006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.377970 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj"] Oct 30 21:47:16.382170 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:47:16.382130 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7673c918_35b4_4784_8814_ab2b57942bf6.slice/crio-17e2d042cd85325e9972585388f84b90f0bcba9cf543319f2e1423a1b560036b WatchSource:0}: Error finding container 17e2d042cd85325e9972585388f84b90f0bcba9cf543319f2e1423a1b560036b: Status 404 returned error can't find the container with id 17e2d042cd85325e9972585388f84b90f0bcba9cf543319f2e1423a1b560036b Oct 30 21:47:16.797516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.797473 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" event={"ID":"7673c918-35b4-4784-8814-ab2b57942bf6","Type":"ContainerStarted","Data":"9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626"} Oct 30 21:47:16.797700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.797568 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" event={"ID":"7673c918-35b4-4784-8814-ab2b57942bf6","Type":"ContainerStarted","Data":"17e2d042cd85325e9972585388f84b90f0bcba9cf543319f2e1423a1b560036b"} Oct 30 21:47:16.797866 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.797835 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:16.826876 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.826825 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" podStartSLOduration=1.826809946 podStartE2EDuration="1.826809946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:16.821487128 +0000 UTC m=+1099.017178248" watchObservedRunningTime="2026-10-30 21:47:16.826809946 +0000 UTC m=+1099.022501076" Oct 30 21:47:16.968983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.964651 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7339-7b7827"] Oct 30 21:47:16.980333 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.980296 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:47:16.980871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:16.980834 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:47:17.026267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.026180 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7339-7b7827"] Oct 30 21:47:17.026763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.026323 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:17.062025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.060700 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l4j2n\"" Oct 30 21:47:17.250830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.246869 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7n4\" (UniqueName: \"kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4\") pod \"test-curl-7339-7b7827\" (UID: \"4f0e6627-6ad4-467f-b615-5bd9d26e81ae\") " pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:17.463037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.456227 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7n4\" (UniqueName: \"kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4\") pod \"test-curl-7339-7b7827\" (UID: \"4f0e6627-6ad4-467f-b615-5bd9d26e81ae\") " pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:17.501148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.500404 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7n4\" (UniqueName: \"kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4\") pod \"test-curl-7339-7b7827\" (UID: \"4f0e6627-6ad4-467f-b615-5bd9d26e81ae\") " pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:17.766035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.765192 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:17.979259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.979198 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:47:17.979259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:17.979252 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:47:18.213972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:18.185371 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7339-7b7827"] Oct 30 21:47:18.875026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:18.868646 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7339-7b7827" event={"ID":"4f0e6627-6ad4-467f-b615-5bd9d26e81ae","Type":"ContainerStarted","Data":"61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a"} Oct 30 21:47:18.875026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:18.868705 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7339-7b7827" event={"ID":"4f0e6627-6ad4-467f-b615-5bd9d26e81ae","Type":"ContainerStarted","Data":"adcc62c1a5c7848ed3c2bde8367d12a6907d15b91ed31174b99e440aacddfa5d"} Oct 30 21:47:18.991973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:18.985634 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:47:18.991973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:18.985689 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 21:47:19.770234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:19.765154 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7339-7b7827" podStartSLOduration=3.7651304789999998 podStartE2EDuration="3.765130479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:18.910317193 +0000 UTC m=+1101.106008315" watchObservedRunningTime="2026-10-30 21:47:19.765130479 +0000 UTC m=+1101.960821598" Oct 30 21:47:19.770234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:19.766885 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7339-7b7827"] Oct 30 21:47:19.775328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:19.773773 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7339-7b7827"] Oct 30 21:47:19.982138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:19.982097 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:19.982604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:19.982584 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:19.983500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:19.983479 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:47:20.886681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:20.886635 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7339-7b7827" podUID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" containerName="test-curl-7339-7b7827" containerID="cri-o://61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a" gracePeriod=2 Oct 30 21:47:23.121078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.121051 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:23.158897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.158837 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4\" (UniqueName: \"kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4\") pod \"4f0e6627-6ad4-467f-b615-5bd9d26e81ae\" (UID: \"4f0e6627-6ad4-467f-b615-5bd9d26e81ae\") " Oct 30 21:47:23.161630 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.161606 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4" pod "4f0e6627-6ad4-467f-b615-5bd9d26e81ae" (UID: "4f0e6627-6ad4-467f-b615-5bd9d26e81ae"). InnerVolumeSpecName "kube-api-access-ll7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:23.261871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.261841 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7n4\" (UniqueName: \"kubernetes.io/projected/4f0e6627-6ad4-467f-b615-5bd9d26e81ae-kube-api-access-ll7n4\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:23.914888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.914853 6808 generic.go:356] "Generic (PLEG): container finished" podID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" containerID="61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a" exitCode=137 Oct 30 21:47:23.915126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.914891 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:23.915126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.914916 6808 scope.go:122] "RemoveContainer" containerID="61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a" Oct 30 21:47:23.927437 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:47:23.927406 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-7b7827\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" pod="opendatahub/test-curl-7339-7b7827" Oct 30 21:47:23.927675 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.927656 6808 scope.go:122] "RemoveContainer" containerID="61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a" Oct 30 21:47:23.928008 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:47:23.927979 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a\": container with ID starting with 61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a not found: ID does not exist" containerID="61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a" Oct 30 21:47:23.928098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.928019 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a"} err="failed to get container status \"61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a\": rpc error: code = NotFound desc = could not find container \"61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a\": container with ID starting with 61ada78b32919231667ac934aa6ab78bc0ef002f6cd182e742646e16a808a85a not found: ID does not exist" Oct 30 21:47:23.933420 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:23.933395 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" path="/var/lib/kubelet/pods/4f0e6627-6ad4-467f-b615-5bd9d26e81ae/volumes" Oct 30 21:47:27.811882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:27.811854 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:47:29.200520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.200488 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx"] Oct 30 21:47:29.201732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.201691 6808 cpu_manager.go:414] "containerMap: removing container" podUID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" containerName="test-curl-7339-7b7827" Oct 30 21:47:29.201912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.201892 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" containerName="test-curl-7339-7b7827" Oct 30 21:47:29.202208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.202191 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0e6627-6ad4-467f-b615-5bd9d26e81ae" containerName="test-curl-7339-7b7827" Oct 30 21:47:29.207846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.207827 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.221916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.221849 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dd5bf8-openshift-default-dockercfg-9bjkm\"" Oct 30 21:47:29.231188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.227187 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx"] Oct 30 21:47:29.368614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.368565 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.375193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.375162 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.375639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.375602 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cql5p\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.383087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.383042 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.392184 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.392156 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.392454 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.392438 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.400124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.400072 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.409994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.400358 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.457279 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.457176 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.585970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.564762 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.586476 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.586442 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.598942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.586657 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.599355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.599324 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.618372 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.618337 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.618666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.618642 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.618867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.618852 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.630285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.630252 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cql5p\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.745108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.745029 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.759055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.665289 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.759294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.612711 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.759403 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.665492 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.759488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.612126 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.759570 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.753776 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.765983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.760035 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.788830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.788800 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.827433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.823265 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:29.877747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:29.877708 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cql5p\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p\") pod \"e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:30.157970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:30.143332 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:30.715000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:30.674865 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn"] Oct 30 21:47:30.763312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:30.751088 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw"] Oct 30 21:47:31.379920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:31.345545 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:31.380404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:31.380366 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:31.383070 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:31.380563 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:47:31.403984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:31.392940 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx"] Oct 30 21:47:32.255357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:32.255312 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" event={"ID":"333f1d92-0c31-4a66-8efb-3fe2c2abd71d","Type":"ContainerStarted","Data":"9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa"} Oct 30 21:47:32.255859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:32.255383 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" event={"ID":"333f1d92-0c31-4a66-8efb-3fe2c2abd71d","Type":"ContainerStarted","Data":"1f450b9f667b12e259225c4c753d9aac282574af7efb8af64a9ce19333fe0aae"} Oct 30 21:47:32.284374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:32.284315 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podStartSLOduration=3.284296194 podStartE2EDuration="3.284296194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:32.282916644 +0000 UTC m=+1114.478607764" watchObservedRunningTime="2026-10-30 21:47:32.284296194 +0000 UTC m=+1114.479987314" Oct 30 21:47:33.158973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:33.154314 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:33.180693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:33.175093 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:33.264018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:33.263990 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:33.264849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:33.264832 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:47:35.675564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:35.675476 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerName="payload-processing" containerID="cri-o://7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91" gracePeriod=25 Oct 30 21:47:35.700621 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:47:35.700590 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:47:35.752222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:35.752180 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" podUID="959951b5-b898-461e-a527-22c8f28f1b66" containerName="payload-pre-processing" containerID="cri-o://1895c4d1a68401dd1d867c1eae3ff57b39369ffaf172ac94cce41c86aa6942a4" gracePeriod=25 Oct 30 21:47:37.653972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:37.653926 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" podUID="959951b5-b898-461e-a527-22c8f28f1b66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:47:37.655075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:37.654163 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:47:47.722182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:47.722122 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:47:47.767979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:47.723040 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" podUID="959951b5-b898-461e-a527-22c8f28f1b66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:47:49.792085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:49.791595 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:49.792085 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:49.791665 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.706982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:50.702103 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:47:50.718697 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:50.718189 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.768064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:50.761071 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.945197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:50.940465 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:47:50.954405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:50.954367 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.040105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.040004 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 30 21:47:51.058756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.040317 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:47:51.260080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.260031 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.566060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.563218 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.714973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.706173 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.736045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.707306 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:47:51.736045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.729656 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:47:51.736045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.730036 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.736045 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.730501 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:47:51.778402 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.776202 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:47:51.778402 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:51.776884 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:47:52.008090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.001999 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:47:52.008090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.002384 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded" Oct 30 21:47:52.103441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.087723 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:52.306025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.305920 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:47:52.336120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.336084 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:47:52.336307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.336138 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:47:52.336307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.336171 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:47:52.351757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.307227 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:52.510401 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.510368 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:47:52.580319 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.580224 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:47:52.655884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.655831 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:52.711556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.707037 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:47:52.711556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.707622 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:47:52.760978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.759506 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:47:52.804121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.804077 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:47:52.807268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:52.807249 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:47:56.168688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.168656 6808 generic.go:356] "Generic (PLEG): container finished" podID="959951b5-b898-461e-a527-22c8f28f1b66" containerID="1895c4d1a68401dd1d867c1eae3ff57b39369ffaf172ac94cce41c86aa6942a4" exitCode=0 Oct 30 21:47:56.169094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.168693 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:47:56.169094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.168812 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" event={"ID":"959951b5-b898-461e-a527-22c8f28f1b66","Type":"ContainerDied","Data":"1895c4d1a68401dd1d867c1eae3ff57b39369ffaf172ac94cce41c86aa6942a4"} Oct 30 21:47:56.170626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.170604 6808 generic.go:356] "Generic (PLEG): container finished" podID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerID="7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91" exitCode=0 Oct 30 21:47:56.170823 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.170730 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" event={"ID":"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72","Type":"ContainerDied","Data":"7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91"} Oct 30 21:47:56.170895 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.170862 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" event={"ID":"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72","Type":"ContainerDied","Data":"bb19dc8100fbdc7aa8f6c9651295b6d60df3fe62fb8f19e0a1e313e1d4a69932"} Oct 30 21:47:56.170940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.170790 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn" Oct 30 21:47:56.171009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.170990 6808 scope.go:122] "RemoveContainer" containerID="7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91" Oct 30 21:47:56.189083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.189024 6808 scope.go:122] "RemoveContainer" containerID="7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91" Oct 30 21:47:56.189822 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:47:56.189779 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91\": container with ID starting with 7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91 not found: ID does not exist" containerID="7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91" Oct 30 21:47:56.190018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.189993 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91"} err="failed to get container status \"7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91\": rpc error: code = NotFound desc = could not find container \"7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91\": container with ID starting with 7d95c12988d3a48e1b8e5970717729692e782fb7039ce69f7c10bcc0eb9afa91 not found: ID does not exist" Oct 30 21:47:56.215297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.215260 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:47:56.408971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.352638 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume\") pod \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " Oct 30 21:47:56.549028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.534167 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume\") pod \"959951b5-b898-461e-a527-22c8f28f1b66\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " Oct 30 21:47:56.630148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.619594 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894\" (UniqueName: \"kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894\") pod \"959951b5-b898-461e-a527-22c8f28f1b66\" (UID: \"959951b5-b898-461e-a527-22c8f28f1b66\") " Oct 30 21:47:56.630450 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.630425 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8\" (UniqueName: \"kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8\") pod \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\" (UID: \"82fad2e0-8fd0-45d7-bf79-b14a4abe9f72\") " Oct 30 21:47:56.632017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.557269 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume" pod "82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" (UID: "82fad2e0-8fd0-45d7-bf79-b14a4abe9f72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:56.632204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.588168 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume" pod "959951b5-b898-461e-a527-22c8f28f1b66" (UID: "959951b5-b898-461e-a527-22c8f28f1b66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:56.642584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.641767 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8" pod "82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" (UID: "82fad2e0-8fd0-45d7-bf79-b14a4abe9f72"). InnerVolumeSpecName "kube-api-access-7xxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:56.652078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.650918 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xxt8\" (UniqueName: \"kubernetes.io/projected/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-kube-api-access-7xxt8\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:56.652078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.650971 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:56.652078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.650986 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959951b5-b898-461e-a527-22c8f28f1b66-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:56.670989 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.663117 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894" pod "959951b5-b898-461e-a527-22c8f28f1b66" (UID: "959951b5-b898-461e-a527-22c8f28f1b66"). InnerVolumeSpecName "kube-api-access-ns894". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:56.753341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.753282 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns894\" (UniqueName: \"kubernetes.io/projected/959951b5-b898-461e-a527-22c8f28f1b66-kube-api-access-ns894\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:56.806861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.806791 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn"] Oct 30 21:47:56.810498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:56.810473 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-ppjbn"] Oct 30 21:47:57.181052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:57.181021 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" Oct 30 21:47:57.181052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:57.181040 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw" event={"ID":"959951b5-b898-461e-a527-22c8f28f1b66","Type":"ContainerDied","Data":"265b0439157b18db1d5ee99d0701c5f79e754b13e1502601abd37e00b9e61498"} Oct 30 21:47:57.181595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:57.181106 6808 scope.go:122] "RemoveContainer" containerID="1895c4d1a68401dd1d867c1eae3ff57b39369ffaf172ac94cce41c86aa6942a4" Oct 30 21:47:57.215649 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:57.215517 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw"] Oct 30 21:47:57.219983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:57.219957 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-bf6pw"] Oct 30 21:47:58.086982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:58.086634 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" path="/var/lib/kubelet/pods/82fad2e0-8fd0-45d7-bf79-b14a4abe9f72/volumes" Oct 30 21:47:58.112970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:47:58.112922 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959951b5-b898-461e-a527-22c8f28f1b66" path="/var/lib/kubelet/pods/959951b5-b898-461e-a527-22c8f28f1b66/volumes" Oct 30 21:48:03.974335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:03.974247 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm"] Oct 30 21:48:03.975683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:03.975627 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" podUID="5e455637-ae8f-461b-8ccd-daca7dab347f" containerName="maas-api" containerID="cri-o://ffef59ac6613bfe144944a8d60291e192880e8aa90e746df0a606f0733ba31d3" gracePeriod=30 Oct 30 21:48:04.509474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:04.509430 6808 generic.go:356] "Generic (PLEG): container finished" podID="5e455637-ae8f-461b-8ccd-daca7dab347f" containerID="ffef59ac6613bfe144944a8d60291e192880e8aa90e746df0a606f0733ba31d3" exitCode=0 Oct 30 21:48:04.509654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:04.509545 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" event={"ID":"5e455637-ae8f-461b-8ccd-daca7dab347f","Type":"ContainerDied","Data":"ffef59ac6613bfe144944a8d60291e192880e8aa90e746df0a606f0733ba31d3"} Oct 30 21:48:04.960021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:04.953076 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:48:05.144929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.144896 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca\") pod \"5e455637-ae8f-461b-8ccd-daca7dab347f\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " Oct 30 21:48:05.145436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.145204 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs\") pod \"5e455637-ae8f-461b-8ccd-daca7dab347f\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " Oct 30 21:48:05.145436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.145302 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\") pod \"5e455637-ae8f-461b-8ccd-daca7dab347f\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " Oct 30 21:48:05.145436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.145328 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca" pod "5e455637-ae8f-461b-8ccd-daca7dab347f" (UID: "5e455637-ae8f-461b-8ccd-daca7dab347f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:05.145436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.145390 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b\" (UniqueName: \"kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b\") pod \"5e455637-ae8f-461b-8ccd-daca7dab347f\" (UID: \"5e455637-ae8f-461b-8ccd-daca7dab347f\") " Oct 30 21:48:05.146500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.146472 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e455637-ae8f-461b-8ccd-daca7dab347f-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:05.147551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.147523 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls" pod "5e455637-ae8f-461b-8ccd-daca7dab347f" (UID: "5e455637-ae8f-461b-8ccd-daca7dab347f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:05.147671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.147612 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b" pod "5e455637-ae8f-461b-8ccd-daca7dab347f" (UID: "5e455637-ae8f-461b-8ccd-daca7dab347f"). InnerVolumeSpecName "kube-api-access-ww29b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:05.147671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.147636 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs" pod "5e455637-ae8f-461b-8ccd-daca7dab347f" (UID: "5e455637-ae8f-461b-8ccd-daca7dab347f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:05.268484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.258628 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:05.268484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.258673 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww29b\" (UniqueName: \"kubernetes.io/projected/5e455637-ae8f-461b-8ccd-daca7dab347f-kube-api-access-ww29b\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:05.268484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.258689 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e455637-ae8f-461b-8ccd-daca7dab347f-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:05.580963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.571732 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" event={"ID":"5e455637-ae8f-461b-8ccd-daca7dab347f","Type":"ContainerDied","Data":"007763ad1ebe7473092e8a720ed4691bcca68874a2ad8424413bf23051e6d641"} Oct 30 21:48:05.580963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.571807 6808 scope.go:122] "RemoveContainer" containerID="ffef59ac6613bfe144944a8d60291e192880e8aa90e746df0a606f0733ba31d3" Oct 30 21:48:05.580963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.571994 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm" Oct 30 21:48:05.722017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.712143 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm"] Oct 30 21:48:05.722017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.717820 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-gpqzm"] Oct 30 21:48:05.758644 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:05.758610 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:48:06.003001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:05.993360 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e455637-ae8f-461b-8ccd-daca7dab347f" path="/var/lib/kubelet/pods/5e455637-ae8f-461b-8ccd-daca7dab347f/volumes" Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.911592 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf"] Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.913047 6808 cpu_manager.go:414] "containerMap: removing container" podUID="5e455637-ae8f-461b-8ccd-daca7dab347f" containerName="maas-api" Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.913069 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e455637-ae8f-461b-8ccd-daca7dab347f" containerName="maas-api" Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.913097 6808 cpu_manager.go:414] "containerMap: removing container" podUID="959951b5-b898-461e-a527-22c8f28f1b66" containerName="payload-pre-processing" Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.913105 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959951b5-b898-461e-a527-22c8f28f1b66" containerName="payload-pre-processing" Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.913132 6808 cpu_manager.go:414] "containerMap: removing container" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerName="payload-processing" Oct 30 21:48:11.914973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.913139 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerName="payload-processing" Oct 30 21:48:11.938003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.933175 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e455637-ae8f-461b-8ccd-daca7dab347f" containerName="maas-api" Oct 30 21:48:11.938003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.933211 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="959951b5-b898-461e-a527-22c8f28f1b66" containerName="payload-pre-processing" Oct 30 21:48:11.938003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.933227 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fad2e0-8fd0-45d7-bf79-b14a4abe9f72" containerName="payload-processing" Oct 30 21:48:11.993971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:11.992388 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.290481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.233505 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.290481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.233673 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2j6m\" (UniqueName: \"kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.290481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.233828 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.290481 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.233900 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.306827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.297646 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2ba0bd-serving-cert\"" Oct 30 21:48:12.306827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.300444 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf"] Oct 30 21:48:12.396251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.396219 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2j6m\" (UniqueName: \"kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.396917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.396892 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.397124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.397108 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.398575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.398445 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.400719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.400698 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.494983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.485293 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2j6m\" (UniqueName: \"kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.586201 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.586024 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:12.619973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:12.594499 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs\") pod \"maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:13.029900 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.005884 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:13.147533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.118660 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz"] Oct 30 21:48:13.187503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.187473 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.188429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.187565 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz"] Oct 30 21:48:13.220177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.220136 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08aafeac\"" Oct 30 21:48:13.246228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.246198 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08aafeac-dockercfg-v7b48\"" Oct 30 21:48:13.300427 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.300164 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.301253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.301228 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvk5\" (UniqueName: \"kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.418975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.413653 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.418975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.413707 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvk5\" (UniqueName: \"kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.418975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.415256 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.496993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.495731 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvk5\" (UniqueName: \"kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5\") pod \"payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:13.767236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:13.767099 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:14.032016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.030492 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:14.224426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.223784 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf"] Oct 30 21:48:14.229313 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:48:14.229265 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf2d7da_0533_4ea8_ae0e_1de61082c5a6.slice/crio-cb336d161c90141ad667648abdc9d7b9b149dbe46bd7d7963e7a82cdf5219c79 WatchSource:0}: Error finding container cb336d161c90141ad667648abdc9d7b9b149dbe46bd7d7963e7a82cdf5219c79: Status 404 returned error can't find the container with id cb336d161c90141ad667648abdc9d7b9b149dbe46bd7d7963e7a82cdf5219c79 Oct 30 21:48:14.232022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.231442 6808 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:48:14.659514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.652799 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz"] Oct 30 21:48:14.836456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.836020 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" event={"ID":"0201a121-f31a-41b4-9e87-3edd447c1e4a","Type":"ContainerStarted","Data":"c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef"} Oct 30 21:48:14.836456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.836099 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" event={"ID":"0201a121-f31a-41b4-9e87-3edd447c1e4a","Type":"ContainerStarted","Data":"1295642f4f5dd07e762d9f0486b354e1d5de661120f254abf5aa1191286b3903"} Oct 30 21:48:14.836456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.836155 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:14.837718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.837689 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" event={"ID":"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6","Type":"ContainerStarted","Data":"cb336d161c90141ad667648abdc9d7b9b149dbe46bd7d7963e7a82cdf5219c79"} Oct 30 21:48:14.860585 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:14.860538 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" podStartSLOduration=2.860524104 podStartE2EDuration="2.860524104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.857478804 +0000 UTC m=+1157.053169923" watchObservedRunningTime="2026-10-30 21:48:14.860524104 +0000 UTC m=+1157.056215222" Oct 30 21:48:15.847435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:15.847389 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" event={"ID":"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6","Type":"ContainerStarted","Data":"49e797500efca3688a5be75f8bc1974528a2b0bbdf6b6f90bfbdef1676193777"} Oct 30 21:48:15.848006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:15.847500 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:15.877144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:15.877099 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" podStartSLOduration=4.33616487 podStartE2EDuration="4.877086609s" totalImagesPullingTime="540.921739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:15.874188259 +0000 UTC m=+1158.069879379" watchObservedRunningTime="2026-10-30 21:48:15.877086609 +0000 UTC m=+1158.072777766" Oct 30 21:48:21.861653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:21.861621 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:48:23.507901 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.507862 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv"] Oct 30 21:48:23.520230 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.520201 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.530995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.524610 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv"] Oct 30 21:48:23.530995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.529027 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08aafeac-serving-cert\"" Oct 30 21:48:23.583988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.583957 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2zj\" (UniqueName: \"kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.584185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.584117 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.584185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.584181 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.584303 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.584250 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.687727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.687357 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.688091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.688064 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.688333 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.688310 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.688710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.688681 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2zj\" (UniqueName: \"kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.690705 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.690660 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.691697 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:23.690916 6808 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-serving-cert: secret "maas-api-e2e-mt-08aafeac-serving-cert" not found Oct 30 21:48:23.691907 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:23.691894 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls podName:cee00dab-e07c-4bb0-9b5a-7647833675d0 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:24.191871837 +0000 UTC m=+1166.387562941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls") pod "maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" (UID: "cee00dab-e07c-4bb0-9b5a-7647833675d0") : secret "maas-api-e2e-mt-08aafeac-serving-cert" not found Oct 30 21:48:23.727257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.727219 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2zj\" (UniqueName: \"kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:23.729732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:23.729680 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:24.204200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.204158 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:24.235199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.235169 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\") pod \"maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:24.468028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.464462 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:24.868626 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:48:24.868579 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee00dab_e07c_4bb0_9b5a_7647833675d0.slice/crio-5919dea25b9cfb15e48ecceb879847fb671fbc902c77f0ce469f1f1f4d62f3df WatchSource:0}: Error finding container 5919dea25b9cfb15e48ecceb879847fb671fbc902c77f0ce469f1f1f4d62f3df: Status 404 returned error can't find the container with id 5919dea25b9cfb15e48ecceb879847fb671fbc902c77f0ce469f1f1f4d62f3df Oct 30 21:48:24.871175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.870887 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv"] Oct 30 21:48:24.970231 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.970197 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm"] Oct 30 21:48:24.970452 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.970426 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" containerID="cri-o://5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6" gracePeriod=30 Oct 30 21:48:24.981628 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.981146 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl"] Oct 30 21:48:24.981628 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.981371 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" containerID="cri-o://1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33" gracePeriod=30 Oct 30 21:48:24.989378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.989333 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b"] Oct 30 21:48:24.989786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:24.989734 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" containerID="cri-o://6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd" gracePeriod=30 Oct 30 21:48:25.002724 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:25.002698 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" event={"ID":"cee00dab-e07c-4bb0-9b5a-7647833675d0","Type":"ContainerStarted","Data":"5919dea25b9cfb15e48ecceb879847fb671fbc902c77f0ce469f1f1f4d62f3df"} Oct 30 21:48:25.851310 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:25.851282 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:48:26.012015 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:26.011973 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" event={"ID":"cee00dab-e07c-4bb0-9b5a-7647833675d0","Type":"ContainerStarted","Data":"d553f3f64e9fc3b3684c7170390613b882562380e2f2da00d92f1698180b12ea"} Oct 30 21:48:26.012449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:26.012067 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:26.036288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:26.036243 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" podStartSLOduration=2.6012281269999997 podStartE2EDuration="3.036230856s" totalImagesPullingTime="435.002729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:26.033120567 +0000 UTC m=+1168.228811687" watchObservedRunningTime="2026-10-30 21:48:26.036230856 +0000 UTC m=+1168.231921974" Oct 30 21:48:30.445010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.444983 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:48:30.536037 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.536007 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:48:30.583277 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.583250 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:48:30.587165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.587137 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.587383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.587366 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.587511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.587409 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.587511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.587446 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.587511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.587499 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.587910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.587887 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.588092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.588074 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.588193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.588178 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.588284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.588208 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert\") pod \"415506c4-eeaf-407f-8562-bd837602183d\" (UID: \"415506c4-eeaf-407f-8562-bd837602183d\") " Oct 30 21:48:30.589283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.589259 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:30.590795 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.590771 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.590990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.590934 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.591586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.591561 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.591680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.591591 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.591680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.591605 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415506c4-eeaf-407f-8562-bd837602183d-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.591859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.591840 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.592156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.592032 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.596011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.593404 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.596011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.594747 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "kube-api-access-g5hzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.596011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.594917 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:30.596011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.595130 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy" pod "415506c4-eeaf-407f-8562-bd837602183d" (UID: "415506c4-eeaf-407f-8562-bd837602183d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.692224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692119 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.692224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692198 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.692448 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692226 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.692448 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692275 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.692448 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692300 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.692614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692450 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.692614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692487 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.692614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692519 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.692614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692549 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.692614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692544 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.692614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692568 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.692916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.692875 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.693083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693028 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.693149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693128 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.693185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693142 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.693221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693181 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.693256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693241 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.693311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693296 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.693389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693373 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.693445 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693409 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.693508 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693491 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.693564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693518 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.693610 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693558 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert\") pod \"408895cb-af66-4aa0-9c06-9074d2657bae\" (UID: \"408895cb-af66-4aa0-9c06-9074d2657bae\") " Oct 30 21:48:30.693699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.693682 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert\") pod \"1fce4081-a70c-4873-8ef6-de2a95d44def\" (UID: \"1fce4081-a70c-4873-8ef6-de2a95d44def\") " Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694712 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694741 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hzv\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-kube-api-access-g5hzv\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694757 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415506c4-eeaf-407f-8562-bd837602183d-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694772 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694785 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694803 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694817 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694831 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415506c4-eeaf-407f-8562-bd837602183d-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694847 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694863 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694878 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415506c4-eeaf-407f-8562-bd837602183d-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.694893 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.695828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.695644 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.695892 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.695869 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.695892 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.695881 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.696148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.696128 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:30.696240 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.696171 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:30.696527 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.696495 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:30.696600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.696563 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.696600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.696591 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "kube-api-access-8wcm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.696739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.696714 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token" pod "408895cb-af66-4aa0-9c06-9074d2657bae" (UID: "408895cb-af66-4aa0-9c06-9074d2657bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.697276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.697256 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.697854 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.697836 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "kube-api-access-5hdfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.698285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.698262 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo" pod "1fce4081-a70c-4873-8ef6-de2a95d44def" (UID: "1fce4081-a70c-4873-8ef6-de2a95d44def"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:30.796803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796770 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.796803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796797 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.796803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796808 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdfn\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-kube-api-access-5hdfn\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796819 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796829 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408895cb-af66-4aa0-9c06-9074d2657bae-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796837 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fce4081-a70c-4873-8ef6-de2a95d44def-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796847 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wcm7\" (UniqueName: \"kubernetes.io/projected/408895cb-af66-4aa0-9c06-9074d2657bae-kube-api-access-8wcm7\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796856 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408895cb-af66-4aa0-9c06-9074d2657bae-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796865 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796875 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796883 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fce4081-a70c-4873-8ef6-de2a95d44def-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.797080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:30.796891 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408895cb-af66-4aa0-9c06-9074d2657bae-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.168096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.167064 6808 generic.go:356] "Generic (PLEG): container finished" podID="415506c4-eeaf-407f-8562-bd837602183d" containerID="5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6" exitCode=0 Oct 30 21:48:31.168096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.167158 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" event={"ID":"415506c4-eeaf-407f-8562-bd837602183d","Type":"ContainerDied","Data":"5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6"} Oct 30 21:48:31.168096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.167208 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" event={"ID":"415506c4-eeaf-407f-8562-bd837602183d","Type":"ContainerDied","Data":"969336d1338d7ab77b3a10f0dadb940fd478f7c0b98c1128df6630351a16c763"} Oct 30 21:48:31.168096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.167237 6808 scope.go:122] "RemoveContainer" containerID="5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6" Oct 30 21:48:31.168096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.167403 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm" Oct 30 21:48:31.199301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.195443 6808 generic.go:356] "Generic (PLEG): container finished" podID="408895cb-af66-4aa0-9c06-9074d2657bae" containerID="6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd" exitCode=0 Oct 30 21:48:31.199301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.195588 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" event={"ID":"408895cb-af66-4aa0-9c06-9074d2657bae","Type":"ContainerDied","Data":"6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd"} Oct 30 21:48:31.199301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.195639 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" event={"ID":"408895cb-af66-4aa0-9c06-9074d2657bae","Type":"ContainerDied","Data":"40a29c6f25304e2d769e8b80526c66550f05d8a953f6741f9a68de8401fa23f0"} Oct 30 21:48:31.199301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.195715 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b" Oct 30 21:48:31.220034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.214719 6808 generic.go:356] "Generic (PLEG): container finished" podID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerID="1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33" exitCode=0 Oct 30 21:48:31.220034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.215022 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" Oct 30 21:48:31.220034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.218772 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" event={"ID":"1fce4081-a70c-4873-8ef6-de2a95d44def","Type":"ContainerDied","Data":"1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33"} Oct 30 21:48:31.220034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.218845 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" event={"ID":"1fce4081-a70c-4873-8ef6-de2a95d44def","Type":"ContainerDied","Data":"032e1aa70d03ffc256d94ed3d282c004c9bf8f1e739402faab2255112881c424"} Oct 30 21:48:31.261311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.261282 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b"] Oct 30 21:48:31.266061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.265999 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-rgc2b"] Oct 30 21:48:31.267014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.266995 6808 scope.go:122] "RemoveContainer" containerID="5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6" Oct 30 21:48:31.268516 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:31.267643 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6\": container with ID starting with 5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6 not found: ID does not exist" containerID="5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6" Oct 30 21:48:31.268516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.267684 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6"} err="failed to get container status \"5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6\": rpc error: code = NotFound desc = could not find container \"5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6\": container with ID starting with 5bbbf5fb9fe5e034579df88c7b34bc85dc0f927473bf5666162b38035787a7d6 not found: ID does not exist" Oct 30 21:48:31.268516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.267708 6808 scope.go:122] "RemoveContainer" containerID="6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd" Oct 30 21:48:31.275609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.275587 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm"] Oct 30 21:48:31.281252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.281230 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-tvbqm"] Oct 30 21:48:31.281805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.281791 6808 scope.go:122] "RemoveContainer" containerID="6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd" Oct 30 21:48:31.282181 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:31.282157 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd\": container with ID starting with 6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd not found: ID does not exist" containerID="6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd" Oct 30 21:48:31.282267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.282189 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd"} err="failed to get container status \"6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd\": rpc error: code = NotFound desc = could not find container \"6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd\": container with ID starting with 6d69445dcfa35248c2121c5d3a133ca71cecbee3a6f9a7c9858fa5783d06b5cd not found: ID does not exist" Oct 30 21:48:31.282267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.282211 6808 scope.go:122] "RemoveContainer" containerID="1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33" Oct 30 21:48:31.291243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.291218 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl"] Oct 30 21:48:31.298232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.298206 6808 scope.go:122] "RemoveContainer" containerID="1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33" Oct 30 21:48:31.298541 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:31.298513 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33\": container with ID starting with 1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33 not found: ID does not exist" containerID="1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33" Oct 30 21:48:31.298683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.298650 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33"} err="failed to get container status \"1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33\": rpc error: code = NotFound desc = could not find container \"1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33\": container with ID starting with 1e25bdc958720f566fba894086de4c7d8d1292207a7c769c5f96ae31dc3bab33 not found: ID does not exist" Oct 30 21:48:31.299445 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.299424 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl"] Oct 30 21:48:31.505427 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.505391 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:48:31.505833 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:31.505456 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-zw7tl" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:32.024973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:32.017705 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" path="/var/lib/kubelet/pods/1fce4081-a70c-4873-8ef6-de2a95d44def/volumes" Oct 30 21:48:32.041439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:32.041400 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" path="/var/lib/kubelet/pods/408895cb-af66-4aa0-9c06-9074d2657bae/volumes" Oct 30 21:48:32.068256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:32.068203 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415506c4-eeaf-407f-8562-bd837602183d" path="/var/lib/kubelet/pods/415506c4-eeaf-407f-8562-bd837602183d/volumes" Oct 30 21:48:32.083175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:32.070537 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:48:33.884126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.884093 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t"] Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885570 6808 cpu_manager.go:414] "containerMap: removing container" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885598 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885640 6808 cpu_manager.go:414] "containerMap: removing container" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885649 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885701 6808 cpu_manager.go:414] "containerMap: removing container" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885710 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885920 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fce4081-a70c-4873-8ef6-de2a95d44def" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885940 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="408895cb-af66-4aa0-9c06-9074d2657bae" containerName="istio-proxy" Oct 30 21:48:33.887164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.885970 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="415506c4-eeaf-407f-8562-bd837602183d" containerName="istio-proxy" Oct 30 21:48:33.894810 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.894783 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:33.907669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.907377 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t"] Oct 30 21:48:33.909186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:33.909160 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-950784-serving-cert\"" Oct 30 21:48:34.031974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.015203 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.031974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.015364 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsmb\" (UniqueName: \"kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.031974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.015436 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.031974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.015494 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.121310 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.121153 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.128685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.128643 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.142563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.129378 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.165973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.163429 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsmb\" (UniqueName: \"kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.165973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.156017 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.284681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.284643 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.286547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.286511 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsmb\" (UniqueName: \"kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.288616 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.288588 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls\") pod \"maas-api-e2e-api-b-950784-748cc4875c-q7g6t\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.521260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.521171 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:34.909975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:34.909561 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t"] Oct 30 21:48:34.967973 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:48:34.951316 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adc6e05_55c3_4d6c_9fc6_4bb2567d0b57.slice/crio-0c62581da5df6f94196a5fd408ead7f090d79694f49641232e77f38811c2ffaf WatchSource:0}: Error finding container 0c62581da5df6f94196a5fd408ead7f090d79694f49641232e77f38811c2ffaf: Status 404 returned error can't find the container with id 0c62581da5df6f94196a5fd408ead7f090d79694f49641232e77f38811c2ffaf Oct 30 21:48:35.630741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:35.630704 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" event={"ID":"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57","Type":"ContainerStarted","Data":"897c2d636e09e6743ce622bde8fc07a687711f102fd4f4640f0fc9d543e8a0ca"} Oct 30 21:48:35.630907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:35.630766 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" event={"ID":"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57","Type":"ContainerStarted","Data":"0c62581da5df6f94196a5fd408ead7f090d79694f49641232e77f38811c2ffaf"} Oct 30 21:48:35.630907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:35.630800 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:35.657507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:35.657435 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" podStartSLOduration=2.156843673 podStartE2EDuration="2.657414271s" totalImagesPullingTime="500.570598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:35.654694329 +0000 UTC m=+1177.850385450" watchObservedRunningTime="2026-10-30 21:48:35.657414271 +0000 UTC m=+1177.853105391" Oct 30 21:48:35.723458 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:35.723376 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:48:40.106397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:40.106208 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk"] Oct 30 21:48:40.107141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:40.107108 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" containerID="cri-o://77f6c9bd1297048dedcb39edf29b557fbd9f491cd68bf48a321ddc8aa57eec34" gracePeriod=30 Oct 30 21:48:40.129982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:40.128995 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn"] Oct 30 21:48:40.129982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:40.129226 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" containerID="cri-o://4b1dfe3ef0e4725ded2b69791223c16deea522158aa02298579f0bdd49bcf018" gracePeriod=30 Oct 30 21:48:40.187575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:40.186028 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n"] Oct 30 21:48:40.187575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:40.186277 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" containerID="cri-o://48151609b60b2599d79a060382b22d7bb925f54490013abeb533fefb3dca66b2" gracePeriod=30 Oct 30 21:48:41.695316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:41.695282 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:48:45.199180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:45.197037 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs"] Oct 30 21:48:45.199180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:45.197299 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" containerID="cri-o://84075b4f6cdf0bb7039c12ae5b51bce0f07eddc30119a35501c59d9635b9d985" gracePeriod=30 Oct 30 21:48:46.078140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.053057 6808 generic.go:356] "Generic (PLEG): container finished" podID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerID="48151609b60b2599d79a060382b22d7bb925f54490013abeb533fefb3dca66b2" exitCode=0 Oct 30 21:48:46.078140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.053178 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" event={"ID":"930b3f3b-88d2-4122-8da4-2fc04f449713","Type":"ContainerDied","Data":"48151609b60b2599d79a060382b22d7bb925f54490013abeb533fefb3dca66b2"} Oct 30 21:48:46.078140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.055704 6808 generic.go:356] "Generic (PLEG): container finished" podID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerID="77f6c9bd1297048dedcb39edf29b557fbd9f491cd68bf48a321ddc8aa57eec34" exitCode=0 Oct 30 21:48:46.078140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.055780 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" event={"ID":"b5982a75-71df-4044-bc2c-a331e4a6cd28","Type":"ContainerDied","Data":"77f6c9bd1297048dedcb39edf29b557fbd9f491cd68bf48a321ddc8aa57eec34"} Oct 30 21:48:46.078140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.058164 6808 generic.go:356] "Generic (PLEG): container finished" podID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerID="4b1dfe3ef0e4725ded2b69791223c16deea522158aa02298579f0bdd49bcf018" exitCode=0 Oct 30 21:48:46.078140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.058356 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" event={"ID":"0a7f4129-3d0c-4a2f-803a-344793e06133","Type":"ContainerDied","Data":"4b1dfe3ef0e4725ded2b69791223c16deea522158aa02298579f0bdd49bcf018"} Oct 30 21:48:46.283041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.282243 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:48:46.299283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.299192 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:48:46.327242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.327172 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:48:46.341900 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.341872 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.342058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.341922 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342001 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342073 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342182 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342120 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342189 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.342284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342243 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.342382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342299 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.342382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342333 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.342382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342364 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342391 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.342548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342429 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342478 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342506 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.342548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342515 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.342803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342555 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.342803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342591 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.342803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342628 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.342803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342666 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.342803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342711 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.342803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342768 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.343123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342838 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.343123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342897 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.343123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342941 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.343123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.342986 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.343123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.343102 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy\") pod \"b5982a75-71df-4044-bc2c-a331e4a6cd28\" (UID: \"b5982a75-71df-4044-bc2c-a331e4a6cd28\") " Oct 30 21:48:46.343359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.343137 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket\") pod \"930b3f3b-88d2-4122-8da4-2fc04f449713\" (UID: \"930b3f3b-88d2-4122-8da4-2fc04f449713\") " Oct 30 21:48:46.343359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.343258 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert\") pod \"0a7f4129-3d0c-4a2f-803a-344793e06133\" (UID: \"0a7f4129-3d0c-4a2f-803a-344793e06133\") " Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.343513 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.343733 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.343876 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344256 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344578 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344601 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344616 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344631 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344645 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7f4129-3d0c-4a2f-803a-344793e06133-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.344930 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.345600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.345293 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.348944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.348011 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.348944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.348283 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:46.348944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.348517 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.348944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.348721 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.350513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.349596 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:46.350513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.349908 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:46.350513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350014 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:46.350513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350105 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:46.350513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350258 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.350513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350460 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:46.350939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350557 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.350939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350697 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.350939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.350929 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "kube-api-access-4cbw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:46.351119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.351020 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.351826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.351799 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "kube-api-access-69ngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:46.352703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.352679 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:46.353092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.353061 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy" pod "b5982a75-71df-4044-bc2c-a331e4a6cd28" (UID: "b5982a75-71df-4044-bc2c-a331e4a6cd28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.353734 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.353704 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:46.354592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.354566 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy" pod "930b3f3b-88d2-4122-8da4-2fc04f449713" (UID: "930b3f3b-88d2-4122-8da4-2fc04f449713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:46.356362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.356334 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc" pod "0a7f4129-3d0c-4a2f-803a-344793e06133" (UID: "0a7f4129-3d0c-4a2f-803a-344793e06133"). InnerVolumeSpecName "kube-api-access-k2hqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454812 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454853 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454872 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454886 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454900 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454915 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbw4\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-kube-api-access-4cbw4\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.454930 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hqc\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-kube-api-access-k2hqc\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455039 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455056 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5982a75-71df-4044-bc2c-a331e4a6cd28-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455073 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930b3f3b-88d2-4122-8da4-2fc04f449713-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455088 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455103 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69ngt\" (UniqueName: \"kubernetes.io/projected/b5982a75-71df-4044-bc2c-a331e4a6cd28-kube-api-access-69ngt\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455116 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455130 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455150 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455163 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5982a75-71df-4044-bc2c-a331e4a6cd28-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455177 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455195 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7f4129-3d0c-4a2f-803a-344793e06133-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455208 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455221 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455236 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7f4129-3d0c-4a2f-803a-344793e06133-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:46.455283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:46.455250 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930b3f3b-88d2-4122-8da4-2fc04f449713-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:47.190331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.190293 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf"] Oct 30 21:48:47.191786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.191756 6808 cpu_manager.go:414] "containerMap: removing container" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" Oct 30 21:48:47.191961 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.191931 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" Oct 30 21:48:47.192093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192082 6808 cpu_manager.go:414] "containerMap: removing container" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" Oct 30 21:48:47.192173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192163 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" Oct 30 21:48:47.192274 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192266 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" Oct 30 21:48:47.192347 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192338 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" Oct 30 21:48:47.192626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192612 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" containerName="istio-proxy" Oct 30 21:48:47.192732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192723 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" containerName="istio-proxy" Oct 30 21:48:47.192816 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.192807 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" containerName="istio-proxy" Oct 30 21:48:47.226872 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.226843 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf"] Oct 30 21:48:47.227260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.227225 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.254963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.248339 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f8144d-openshift-default-dockercfg-b5qxh\"" Oct 30 21:48:47.449365 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.449285 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" Oct 30 21:48:47.453094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.452042 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n" event={"ID":"930b3f3b-88d2-4122-8da4-2fc04f449713","Type":"ContainerDied","Data":"39eedf86981bdb6298f2eb0e4ef114e345b3be34ed7b4095ff4b9dc41bcec828"} Oct 30 21:48:47.453094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.452942 6808 scope.go:122] "RemoveContainer" containerID="48151609b60b2599d79a060382b22d7bb925f54490013abeb533fefb3dca66b2" Oct 30 21:48:47.519518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.512790 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" event={"ID":"b5982a75-71df-4044-bc2c-a331e4a6cd28","Type":"ContainerDied","Data":"9857794a25b13adec564dc823a3551a5ae8713761418d77a108236e99903701c"} Oct 30 21:48:47.519518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.512918 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk" Oct 30 21:48:47.522544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.522475 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.522781 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92ch\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.522822 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.522852 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.522992 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.523029 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.523064 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.523288 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.523469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.523318 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.529899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.529859 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" Oct 30 21:48:47.530052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.529907 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn" event={"ID":"0a7f4129-3d0c-4a2f-803a-344793e06133","Type":"ContainerDied","Data":"c3d3fe5ccafdc290e53cab7b94e8c9feb0c6d6d050b66c077e49e42bfa97a651"} Oct 30 21:48:47.549119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.549043 6808 scope.go:122] "RemoveContainer" containerID="77f6c9bd1297048dedcb39edf29b557fbd9f491cd68bf48a321ddc8aa57eec34" Oct 30 21:48:47.584494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.574091 6808 scope.go:122] "RemoveContainer" containerID="4b1dfe3ef0e4725ded2b69791223c16deea522158aa02298579f0bdd49bcf018" Oct 30 21:48:47.584494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.582620 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n"] Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.633006 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.633057 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.633211 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.634263 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92ch\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.634308 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.634344 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.634490 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.634528 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.634972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.634566 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.635656 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.635233 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.635656 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.635478 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.635771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.635694 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.655971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.636280 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.655971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.636904 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.741042 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.740913 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-rvs7n"] Oct 30 21:48:47.776972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.776158 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk"] Oct 30 21:48:47.792976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.792108 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.800051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.799941 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.814499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.814460 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-5tpsk"] Oct 30 21:48:47.817490 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.817064 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.817490 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.817209 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92ch\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:47.830397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.830206 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn"] Oct 30 21:48:47.836970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.836876 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-95bsn"] Oct 30 21:48:47.938793 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.938753 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7f4129-3d0c-4a2f-803a-344793e06133" path="/var/lib/kubelet/pods/0a7f4129-3d0c-4a2f-803a-344793e06133/volumes" Oct 30 21:48:47.939626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.939604 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930b3f3b-88d2-4122-8da4-2fc04f449713" path="/var/lib/kubelet/pods/930b3f3b-88d2-4122-8da4-2fc04f449713/volumes" Oct 30 21:48:47.940309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:47.940289 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5982a75-71df-4044-bc2c-a331e4a6cd28" path="/var/lib/kubelet/pods/b5982a75-71df-4044-bc2c-a331e4a6cd28/volumes" Oct 30 21:48:48.049437 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.049348 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:48.402778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.402609 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf"] Oct 30 21:48:48.404585 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:48:48.404543 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e659ef2_1952_4763_819a_f0034d970c84.slice/crio-08930681bd3a20ceebf8c12d95607b082d49bc590bb43379937e72219ae6f878 WatchSource:0}: Error finding container 08930681bd3a20ceebf8c12d95607b082d49bc590bb43379937e72219ae6f878: Status 404 returned error can't find the container with id 08930681bd3a20ceebf8c12d95607b082d49bc590bb43379937e72219ae6f878 Oct 30 21:48:48.407731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.407691 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:48.407942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.407920 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:48.408103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.408084 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:48.546595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.546561 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" event={"ID":"5e659ef2-1952-4763-819a-f0034d970c84","Type":"ContainerStarted","Data":"883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be"} Oct 30 21:48:48.547095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.546627 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" event={"ID":"5e659ef2-1952-4763-819a-f0034d970c84","Type":"ContainerStarted","Data":"08930681bd3a20ceebf8c12d95607b082d49bc590bb43379937e72219ae6f878"} Oct 30 21:48:48.599721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:48.597990 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podStartSLOduration=2.597970864 podStartE2EDuration="2.597970864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:48.578770212 +0000 UTC m=+1190.774461332" watchObservedRunningTime="2026-10-30 21:48:48.597970864 +0000 UTC m=+1190.793661978" Oct 30 21:48:49.068973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:49.057199 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:49.068973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:49.057433 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:48:49.068973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:49.057512 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:48:49.338196 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:49.338108 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:48:49.338196 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:49.338181 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:49.498369 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:49.498335 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj"] Oct 30 21:48:50.006752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.002483 6808 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:48:50.006752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.002553 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:48:50.083969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.072239 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:50.610276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.609089 6808 generic.go:356] "Generic (PLEG): container finished" podID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerID="84075b4f6cdf0bb7039c12ae5b51bce0f07eddc30119a35501c59d9635b9d985" exitCode=0 Oct 30 21:48:50.610276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.609208 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" event={"ID":"0c01e6b8-5944-4e27-8328-e240f99889cc","Type":"ContainerDied","Data":"84075b4f6cdf0bb7039c12ae5b51bce0f07eddc30119a35501c59d9635b9d985"} Oct 30 21:48:50.610276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.609280 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" event={"ID":"0c01e6b8-5944-4e27-8328-e240f99889cc","Type":"ContainerDied","Data":"b8c2257cb3d33f09e98495c3b7e0eb13659e56f4d2437b20ca6e9ded9af14148"} Oct 30 21:48:50.610276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.609307 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c2257cb3d33f09e98495c3b7e0eb13659e56f4d2437b20ca6e9ded9af14148" Oct 30 21:48:50.611228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.611197 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:48:50.612995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.612104 6808 generic.go:356] "Generic (PLEG): container finished" podID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerID="9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa" exitCode=0 Oct 30 21:48:50.612995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.612878 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" event={"ID":"333f1d92-0c31-4a66-8efb-3fe2c2abd71d","Type":"ContainerDied","Data":"9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa"} Oct 30 21:48:50.612995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.612918 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:50.613438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.613417 6808 scope.go:122] "RemoveContainer" containerID="9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa" Oct 30 21:48:50.614998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.614979 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:48:50.615714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.615687 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:50.615784 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.615753 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:50.615826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.615787 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:48:50.796998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.796897 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.796998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.796985 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797096 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797189 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797231 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797291 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797412 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797504 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.797633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.797609 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token\") pod \"0c01e6b8-5944-4e27-8328-e240f99889cc\" (UID: \"0c01e6b8-5944-4e27-8328-e240f99889cc\") " Oct 30 21:48:50.802749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.802713 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.806360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.804425 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.806360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.804593 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.806360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.805501 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.806360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.805817 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:50.819565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.819533 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:50.821186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.821163 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:50.824408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.824367 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.825943 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.825901 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg" pod "0c01e6b8-5944-4e27-8328-e240f99889cc" (UID: "0c01e6b8-5944-4e27-8328-e240f99889cc"). InnerVolumeSpecName "kube-api-access-rqgsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916839 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916880 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916896 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916910 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916924 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c01e6b8-5944-4e27-8328-e240f99889cc-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916937 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916967 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgsg\" (UniqueName: \"kubernetes.io/projected/0c01e6b8-5944-4e27-8328-e240f99889cc-kube-api-access-rqgsg\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916982 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.920183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.916997 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c01e6b8-5944-4e27-8328-e240f99889cc-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.926514 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.926485 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7339-62b77b"] Oct 30 21:48:50.927339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.927319 6808 cpu_manager.go:414] "containerMap: removing container" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" Oct 30 21:48:50.927339 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.927339 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" Oct 30 21:48:50.927494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.927472 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" containerName="istio-proxy" Oct 30 21:48:50.938930 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.938908 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:50.944087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:50.939783 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7339-62b77b"] Oct 30 21:48:51.021577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.021539 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l4j2n\"" Oct 30 21:48:51.122431 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.122396 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgqk\" (UniqueName: \"kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk\") pod \"test-curl-7339-62b77b\" (UID: \"66b8a568-70aa-4426-945f-db4a250b12b4\") " pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:51.225870 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.225770 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgqk\" (UniqueName: \"kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk\") pod \"test-curl-7339-62b77b\" (UID: \"66b8a568-70aa-4426-945f-db4a250b12b4\") " pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:51.266022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.253986 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgqk\" (UniqueName: \"kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk\") pod \"test-curl-7339-62b77b\" (UID: \"66b8a568-70aa-4426-945f-db4a250b12b4\") " pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:51.332015 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.329517 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:51.761993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.760413 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" event={"ID":"333f1d92-0c31-4a66-8efb-3fe2c2abd71d","Type":"ContainerStarted","Data":"a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935"} Oct 30 21:48:51.765586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.765563 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs" Oct 30 21:48:51.912695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.912654 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs"] Oct 30 21:48:51.930822 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:51.930793 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-khhhs"] Oct 30 21:48:52.001109 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:52.001085 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7339-62b77b"] Oct 30 21:48:52.002189 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:48:52.002155 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66b8a568_70aa_4426_945f_db4a250b12b4.slice/crio-241c8fbaefe20b778dfff30d5b4c16af3eab246797bf61ef53f0d66c1306be68 WatchSource:0}: Error finding container 241c8fbaefe20b778dfff30d5b4c16af3eab246797bf61ef53f0d66c1306be68: Status 404 returned error can't find the container with id 241c8fbaefe20b778dfff30d5b4c16af3eab246797bf61ef53f0d66c1306be68 Oct 30 21:48:52.154053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:52.144323 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:48:52.154053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:52.144477 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:52.154053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:52.144522 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:53.160940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.160903 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:53.176096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.176039 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:53.224530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.224490 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7339-62b77b" event={"ID":"66b8a568-70aa-4426-945f-db4a250b12b4","Type":"ContainerStarted","Data":"56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d"} Oct 30 21:48:53.224776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.224757 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7339-62b77b" event={"ID":"66b8a568-70aa-4426-945f-db4a250b12b4","Type":"ContainerStarted","Data":"241c8fbaefe20b778dfff30d5b4c16af3eab246797bf61ef53f0d66c1306be68"} Oct 30 21:48:53.524148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.524050 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:53.592117 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.592067 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:53.662976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.662820 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:53.823265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.823139 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:53.825622 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.825575 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:53.912401 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.912359 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:53.944089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.944036 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:54.005974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:53.998433 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:54.035066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.033871 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded" Oct 30 21:48:54.057067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.057020 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c01e6b8-5944-4e27-8328-e240f99889cc" path="/var/lib/kubelet/pods/0c01e6b8-5944-4e27-8328-e240f99889cc/volumes" Oct 30 21:48:54.165221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.157830 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:54.178074 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.178034 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:54.334268 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.334058 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:54.514029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.498940 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" podUID="7673c918-35b4-4784-8814-ab2b57942bf6" containerName="payload-pre-processing" containerID="cri-o://9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626" gracePeriod=25 Oct 30 21:48:54.725806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:54.725679 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:55.167059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.167019 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:55.201285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.182985 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:55.327969 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:55.327487 6808 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7673c918_35b4_4784_8814_ab2b57942bf6.slice/crio-conmon-9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7673c918_35b4_4784_8814_ab2b57942bf6.slice/crio-9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:48:55.814305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.814108 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:55.896374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.896335 6808 generic.go:356] "Generic (PLEG): container finished" podID="7673c918-35b4-4784-8814-ab2b57942bf6" containerID="9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626" exitCode=0 Oct 30 21:48:55.896933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.896901 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" event={"ID":"7673c918-35b4-4784-8814-ab2b57942bf6","Type":"ContainerDied","Data":"9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626"} Oct 30 21:48:55.932434 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.932392 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:55.973976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:55.973087 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:56.001173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.001101 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7339-62b77b" podStartSLOduration=6.001081856 podStartE2EDuration="6.001081856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:53.291843836 +0000 UTC m=+1195.487534957" watchObservedRunningTime="2026-10-30 21:48:56.001081856 +0000 UTC m=+1198.196772976" Oct 30 21:48:56.034975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.021538 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7339-62b77b"] Oct 30 21:48:56.036574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.036531 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7339-62b77b" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" containerName="test-curl-7339-62b77b" containerID="cri-o://56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d" gracePeriod=2 Oct 30 21:48:56.038386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.038358 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7339-62b77b"] Oct 30 21:48:56.067085 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:56.042344 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-62b77b\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:56.108056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.099856 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:48:56.139439 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:56.135465 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-62b77b\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:56.168977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.167123 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:56.168977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.167187 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:56.492050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.469591 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume\") pod \"7673c918-35b4-4784-8814-ab2b57942bf6\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " Oct 30 21:48:56.492050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.469670 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz\" (UniqueName: \"kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz\") pod \"7673c918-35b4-4784-8814-ab2b57942bf6\" (UID: \"7673c918-35b4-4784-8814-ab2b57942bf6\") " Oct 30 21:48:56.492050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.472610 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume" pod "7673c918-35b4-4784-8814-ab2b57942bf6" (UID: "7673c918-35b4-4784-8814-ab2b57942bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:56.515020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.514536 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz" pod "7673c918-35b4-4784-8814-ab2b57942bf6" (UID: "7673c918-35b4-4784-8814-ab2b57942bf6"). InnerVolumeSpecName "kube-api-access-xspjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:56.606307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.606269 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673c918-35b4-4784-8814-ab2b57942bf6-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:56.606307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.606303 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xspjz\" (UniqueName: \"kubernetes.io/projected/7673c918-35b4-4784-8814-ab2b57942bf6-kube-api-access-xspjz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:56.926124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.920810 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" event={"ID":"7673c918-35b4-4784-8814-ab2b57942bf6","Type":"ContainerDied","Data":"17e2d042cd85325e9972585388f84b90f0bcba9cf543319f2e1423a1b560036b"} Oct 30 21:48:56.926124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.920908 6808 scope.go:122] "RemoveContainer" containerID="9f652fb3ca34f4bb2ee83a57a51c1abe826eff2043c5f4ce93d0a3cbdd472626" Oct 30 21:48:56.926124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:56.921110 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj" Oct 30 21:48:56.937196 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:56.937154 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-62b77b\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:56.995346 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:56.991334 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-62b77b\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:57.065100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:57.065030 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj"] Oct 30 21:48:57.112971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:57.112712 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd3c4c-7fc6885467-k8vvj"] Oct 30 21:48:57.158331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:57.158287 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:57.158703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:57.158674 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:57.930024 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:57.929938 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-62b77b\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:57.935294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:57.935268 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7673c918-35b4-4784-8814-ab2b57942bf6" path="/var/lib/kubelet/pods/7673c918-35b4-4784-8814-ab2b57942bf6/volumes" Oct 30 21:48:58.174971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.150052 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:58.174971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.150112 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:48:58.584099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.583861 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:58.671747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.671709 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk\" (UniqueName: \"kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk\") pod \"66b8a568-70aa-4426-945f-db4a250b12b4\" (UID: \"66b8a568-70aa-4426-945f-db4a250b12b4\") " Oct 30 21:48:58.676496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.676468 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk" pod "66b8a568-70aa-4426-945f-db4a250b12b4" (UID: "66b8a568-70aa-4426-945f-db4a250b12b4"). InnerVolumeSpecName "kube-api-access-qxgqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:58.775419 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.775379 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgqk\" (UniqueName: \"kubernetes.io/projected/66b8a568-70aa-4426-945f-db4a250b12b4-kube-api-access-qxgqk\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:58.955976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.952895 6808 generic.go:356] "Generic (PLEG): container finished" podID="66b8a568-70aa-4426-945f-db4a250b12b4" containerID="56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d" exitCode=137 Oct 30 21:48:58.955976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.953038 6808 scope.go:122] "RemoveContainer" containerID="56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d" Oct 30 21:48:58.955976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.953149 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:58.989027 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:58.988148 6808 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7339-62b77b\" is forbidden: User \"system:node:ip-10-0-137-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-168.us-east-2.compute.internal' and this object" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" pod="opendatahub/test-curl-7339-62b77b" Oct 30 21:48:58.994040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.994020 6808 scope.go:122] "RemoveContainer" containerID="56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d" Oct 30 21:48:58.994301 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:48:58.994276 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d\": container with ID starting with 56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d not found: ID does not exist" containerID="56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d" Oct 30 21:48:58.994407 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:58.994307 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d"} err="failed to get container status \"56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d\": rpc error: code = NotFound desc = could not find container \"56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d\": container with ID starting with 56816f4184dccf9faef12df81bcd431b3fbb5e9b011d5b017dc810b0bc81715d not found: ID does not exist" Oct 30 21:48:59.155986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:59.154386 6808 patch_prober.go:29] interesting pod/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:59.155986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:48:59.154444 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 21:49:00.000675 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:00.000378 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" path="/var/lib/kubelet/pods/66b8a568-70aa-4426-945f-db4a250b12b4/volumes" Oct 30 21:49:00.149298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:00.149269 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:49:00.149838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:00.149819 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:49:00.150751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:00.150728 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:49:04.083208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:04.083116 6808 generic.go:356] "Generic (PLEG): container finished" podID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerID="8c370170f2eb312ffca0ef4482d9fa8342e81e5965cc3cf17c56935b8ead6714" exitCode=0 Oct 30 21:49:04.083208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:04.083147 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" event={"ID":"f46ac04d-5ae1-43f1-be29-5a08872cb890","Type":"ContainerDied","Data":"8c370170f2eb312ffca0ef4482d9fa8342e81e5965cc3cf17c56935b8ead6714"} Oct 30 21:49:04.083759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:04.083541 6808 scope.go:122] "RemoveContainer" containerID="8c370170f2eb312ffca0ef4482d9fa8342e81e5965cc3cf17c56935b8ead6714" Oct 30 21:49:04.085270 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:04.085240 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:04.085397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:04.085314 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:04.085397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:04.085348 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:05.092896 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:05.092859 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" event={"ID":"f46ac04d-5ae1-43f1-be29-5a08872cb890","Type":"ContainerStarted","Data":"885ae0f9f0bf71f69fee36fcd5afa1f5cce63fa9b474dbeba4b0fa914380065a"} Oct 30 21:49:05.652327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:05.652294 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:49:05.652714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:05.652624 6808 patch_prober.go:29] interesting pod/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:49:05.652891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:05.652865 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:49:05.710250 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:05.710218 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:49:06.390187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:06.390153 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz"] Oct 30 21:49:06.703025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:06.663612 6808 patch_prober.go:29] interesting pod/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:49:06.703025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:06.663674 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:49:07.666779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:07.666744 6808 patch_prober.go:29] interesting pod/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:49:07.667381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:07.667350 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:49:08.660346 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:08.660307 6808 patch_prober.go:29] interesting pod/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:49:08.660637 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:08.660607 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:49:09.495548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.495212 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8"] Oct 30 21:49:09.502451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.500810 6808 cpu_manager.go:414] "containerMap: removing container" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" containerName="test-curl-7339-62b77b" Oct 30 21:49:09.502451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.500841 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" containerName="test-curl-7339-62b77b" Oct 30 21:49:09.502451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.500865 6808 cpu_manager.go:414] "containerMap: removing container" podUID="7673c918-35b4-4784-8814-ab2b57942bf6" containerName="payload-pre-processing" Oct 30 21:49:09.502451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.500873 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7673c918-35b4-4784-8814-ab2b57942bf6" containerName="payload-pre-processing" Oct 30 21:49:09.502451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.501303 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b8a568-70aa-4426-945f-db4a250b12b4" containerName="test-curl-7339-62b77b" Oct 30 21:49:09.502451 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.501323 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="7673c918-35b4-4784-8814-ab2b57942bf6" containerName="payload-pre-processing" Oct 30 21:49:09.505783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.505748 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.510306 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.510267 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8"] Oct 30 21:49:09.511344 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.511307 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f8144d-serving-cert\"" Oct 30 21:49:09.672977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.666787 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.672977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.666942 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.672977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.667160 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.672977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.667249 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc59d\" (UniqueName: \"kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.708981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.705535 6808 patch_prober.go:29] interesting pod/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:49:09.708981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.705595 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:49:09.772029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.768912 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.772029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.769067 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.772029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.769152 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.772029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.769221 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc59d\" (UniqueName: \"kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.772029 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:09.769618 6808 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-serving-cert: secret "maas-api-e2e-api-a-f8144d-serving-cert" not found Oct 30 21:49:09.772029 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:09.769680 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls podName:bfd168fc-9a7b-4dbc-801a-a30cf715fdbc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:10.269660651 +0000 UTC m=+1212.465351761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls") pod "maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" (UID: "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc") : secret "maas-api-e2e-api-a-f8144d-serving-cert" not found Oct 30 21:49:09.805975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.805849 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.867292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.839975 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:09.867292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:09.866269 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc59d\" (UniqueName: \"kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:10.295621 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:10.295571 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:10.299839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:10.299797 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\") pod \"maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:10.439295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:10.439254 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:10.692627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:10.692157 6808 patch_prober.go:29] interesting pod/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:49:10.692627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:10.692221 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 21:49:10.999780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:10.977905 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8"] Oct 30 21:49:11.231770 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:11.231733 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" event={"ID":"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc","Type":"ContainerStarted","Data":"fb7c990027d6067aa39d528f5e208a10ee31a7a00b7f6852038580f8a08ab468"} Oct 30 21:49:11.391658 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:11.391613 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" podUID="0201a121-f31a-41b4-9e87-3edd447c1e4a" containerName="payload-processing" containerID="cri-o://c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef" gracePeriod=25 Oct 30 21:49:11.672097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:11.667693 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:49:11.672097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:11.669354 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:49:11.674325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:11.674307 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:49:11.873138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:11.873112 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:49:12.034029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.033986 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5\" (UniqueName: \"kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5\") pod \"0201a121-f31a-41b4-9e87-3edd447c1e4a\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " Oct 30 21:49:12.034275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.034137 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume\") pod \"0201a121-f31a-41b4-9e87-3edd447c1e4a\" (UID: \"0201a121-f31a-41b4-9e87-3edd447c1e4a\") " Oct 30 21:49:12.034503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.034480 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume" pod "0201a121-f31a-41b4-9e87-3edd447c1e4a" (UID: "0201a121-f31a-41b4-9e87-3edd447c1e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:12.034904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.034886 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0201a121-f31a-41b4-9e87-3edd447c1e4a-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.036205 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.036188 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5" pod "0201a121-f31a-41b4-9e87-3edd447c1e4a" (UID: "0201a121-f31a-41b4-9e87-3edd447c1e4a"). InnerVolumeSpecName "kube-api-access-nnvk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:12.136236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.136201 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnvk5\" (UniqueName: \"kubernetes.io/projected/0201a121-f31a-41b4-9e87-3edd447c1e4a-kube-api-access-nnvk5\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.240562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.240531 6808 generic.go:356] "Generic (PLEG): container finished" podID="0201a121-f31a-41b4-9e87-3edd447c1e4a" containerID="c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef" exitCode=0 Oct 30 21:49:12.240738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.240577 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" Oct 30 21:49:12.240738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.240608 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" event={"ID":"0201a121-f31a-41b4-9e87-3edd447c1e4a","Type":"ContainerDied","Data":"c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef"} Oct 30 21:49:12.240738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.240667 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz" event={"ID":"0201a121-f31a-41b4-9e87-3edd447c1e4a","Type":"ContainerDied","Data":"1295642f4f5dd07e762d9f0486b354e1d5de661120f254abf5aa1191286b3903"} Oct 30 21:49:12.240738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.240696 6808 scope.go:122] "RemoveContainer" containerID="c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef" Oct 30 21:49:12.242725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.242692 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" event={"ID":"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc","Type":"ContainerStarted","Data":"1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3"} Oct 30 21:49:12.253000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.252977 6808 scope.go:122] "RemoveContainer" containerID="c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef" Oct 30 21:49:12.253275 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:12.253257 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef\": container with ID starting with c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef not found: ID does not exist" containerID="c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef" Oct 30 21:49:12.253340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.253282 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef"} err="failed to get container status \"c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef\": rpc error: code = NotFound desc = could not find container \"c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef\": container with ID starting with c8cbe395df8a96ddae86fd95f8f4941a9feae265eac49acebd830ef84361fbef not found: ID does not exist" Oct 30 21:49:12.269041 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.268999 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" podStartSLOduration=2.902633612 podStartE2EDuration="3.268984432s" totalImagesPullingTime="366.35082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:12.266470824 +0000 UTC m=+1214.462161949" watchObservedRunningTime="2026-10-30 21:49:12.268984432 +0000 UTC m=+1214.464675556" Oct 30 21:49:12.363990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.356306 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz"] Oct 30 21:49:12.363990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:12.360866 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08aafeac-6cf6c9fdb9-qbsvz"] Oct 30 21:49:13.254173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:13.254138 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:13.932237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:13.932189 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0201a121-f31a-41b4-9e87-3edd447c1e4a" path="/var/lib/kubelet/pods/0201a121-f31a-41b4-9e87-3edd447c1e4a/volumes" Oct 30 21:49:17.717118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:17.717087 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm"] Oct 30 21:49:17.723110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:17.723084 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q"] Oct 30 21:49:17.732090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:17.732062 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k"] Oct 30 21:49:17.738508 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:17.738475 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd"] Oct 30 21:49:18.544503 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:18.544472 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv"] Oct 30 21:49:18.544729 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:18.544701 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" podUID="cee00dab-e07c-4bb0-9b5a-7647833675d0" containerName="maas-api" containerID="cri-o://d553f3f64e9fc3b3684c7170390613b882562380e2f2da00d92f1698180b12ea" gracePeriod=30 Oct 30 21:49:19.362438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.362112 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:49:19.392062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.391901 6808 generic.go:356] "Generic (PLEG): container finished" podID="cee00dab-e07c-4bb0-9b5a-7647833675d0" containerID="d553f3f64e9fc3b3684c7170390613b882562380e2f2da00d92f1698180b12ea" exitCode=0 Oct 30 21:49:19.392062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.391981 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" event={"ID":"cee00dab-e07c-4bb0-9b5a-7647833675d0","Type":"ContainerDied","Data":"d553f3f64e9fc3b3684c7170390613b882562380e2f2da00d92f1698180b12ea"} Oct 30 21:49:19.511566 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.509106 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:49:19.571096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.571058 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\") pod \"cee00dab-e07c-4bb0-9b5a-7647833675d0\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " Oct 30 21:49:19.571285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.571141 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs\") pod \"cee00dab-e07c-4bb0-9b5a-7647833675d0\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " Oct 30 21:49:19.571355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.571323 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca\") pod \"cee00dab-e07c-4bb0-9b5a-7647833675d0\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " Oct 30 21:49:19.571411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.571379 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj\" (UniqueName: \"kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj\") pod \"cee00dab-e07c-4bb0-9b5a-7647833675d0\" (UID: \"cee00dab-e07c-4bb0-9b5a-7647833675d0\") " Oct 30 21:49:19.571615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.571592 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca" pod "cee00dab-e07c-4bb0-9b5a-7647833675d0" (UID: "cee00dab-e07c-4bb0-9b5a-7647833675d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:19.572273 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.572247 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee00dab-e07c-4bb0-9b5a-7647833675d0-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.576305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.576272 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls" pod "cee00dab-e07c-4bb0-9b5a-7647833675d0" (UID: "cee00dab-e07c-4bb0-9b5a-7647833675d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:19.576305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.576286 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj" pod "cee00dab-e07c-4bb0-9b5a-7647833675d0" (UID: "cee00dab-e07c-4bb0-9b5a-7647833675d0"). InnerVolumeSpecName "kube-api-access-8d2zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:19.576305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.576283 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs" pod "cee00dab-e07c-4bb0-9b5a-7647833675d0" (UID: "cee00dab-e07c-4bb0-9b5a-7647833675d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:19.684521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.683154 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.684521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.683192 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee00dab-e07c-4bb0-9b5a-7647833675d0-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.684521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:19.683207 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2zj\" (UniqueName: \"kubernetes.io/projected/cee00dab-e07c-4bb0-9b5a-7647833675d0-kube-api-access-8d2zj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:20.400483 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:20.400456 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" Oct 30 21:49:20.400981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:20.400480 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv" event={"ID":"cee00dab-e07c-4bb0-9b5a-7647833675d0","Type":"ContainerDied","Data":"5919dea25b9cfb15e48ecceb879847fb671fbc902c77f0ce469f1f1f4d62f3df"} Oct 30 21:49:20.400981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:20.400544 6808 scope.go:122] "RemoveContainer" containerID="d553f3f64e9fc3b3684c7170390613b882562380e2f2da00d92f1698180b12ea" Oct 30 21:49:20.422007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:20.421978 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv"] Oct 30 21:49:20.427863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:20.427836 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08aafeac-5cd46bdfc7-58lcv"] Oct 30 21:49:21.930447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:21.930411 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee00dab-e07c-4bb0-9b5a-7647833675d0" path="/var/lib/kubelet/pods/cee00dab-e07c-4bb0-9b5a-7647833675d0/volumes" Oct 30 21:49:24.407969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:24.404176 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w"] Oct 30 21:49:31.587970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:31.585047 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:49:31.651176 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:31.651108 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:49:31.768972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:31.766710 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:31.887032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:31.881041 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.053971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.047588 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.482075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.424843 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq"] Oct 30 21:49:32.541469 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.541419 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" containerID="cri-o://885ae0f9f0bf71f69fee36fcd5afa1f5cce63fa9b474dbeba4b0fa914380065a" gracePeriod=30 Oct 30 21:49:32.555197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.528240 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.588225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.588175 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.617657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.617626 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:49:32.654609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.654554 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.812028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.725079 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:49:32.849855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.814175 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" containerID="cri-o://145434cc8bc7436f2f98e6f8c210e173a8d638e7be843e61924fe450a13c5c53" gracePeriod=45 Oct 30 21:49:32.849855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.814365 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" containerID="cri-o://8cffa7465e40afcbcab708342325b94b9a3710d5bcc8ba8d4ce2775a5e6c545c" gracePeriod=45 Oct 30 21:49:32.849855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.814449 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" containerID="cri-o://d601ceed50595c9e926212421e2bf814baac0dab4cf68da822879a2a23e00b64" gracePeriod=45 Oct 30 21:49:32.849855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.815990 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:49:32.849855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.846755 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.849855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.847265 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:49:32.922316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.892104 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.922316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.899891 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:49:32.971302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.957857 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:32.971302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:32.963246 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.039905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.001161 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.068379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.049178 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.106971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.087193 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:49:33.275976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.266223 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" containerID="cri-o://4aacb30c4c88a8ba05bcd50e311af5d703d94bfddbeaf422c374f7916311ce57" gracePeriod=45 Oct 30 21:49:33.275976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.267601 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:49:33.672767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.656576 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": EOF" Oct 30 21:49:33.672767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.656695 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:49:33.672767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.657040 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.672767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.657456 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:49:33.709318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.706859 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.796977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.746066 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Oct 30 21:49:33.796977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.751055 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.796977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.786532 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Oct 30 21:49:33.851099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.851048 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.905637 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.905590 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.969724 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.969627 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:49:33.993058 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:33.989704 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.007245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.007201 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.058259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.058217 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.137425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.101274 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": read tcp 10.134.0.2:59158->10.134.0.49:8000: read: connection reset by peer" Oct 30 21:49:34.144256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.101543 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": read tcp 10.134.0.2:58396->10.134.0.56:8000: read: connection reset by peer" Oct 30 21:49:34.265539 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.265455 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:49:34.265888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.265869 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:49:34.266051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.266039 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:49:34.270971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.267809 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.302974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.271660 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:49:34.548541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.548455 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:49:34.696981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.695422 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.705978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.704910 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:49:34.823579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.823471 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 21:49:34.883977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.876166 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Oct 30 21:49:35.153980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.954921 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:35.175008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.980414 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:49:35.183794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.980858 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:35.187192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.982924 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:49:35.195003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:34.983985 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:35.195003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.124551 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 30 21:49:35.195003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.167672 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Oct 30 21:49:35.197746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.197301 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:49:35.241967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.235393 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:49:35.242158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.242028 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:49:35.273573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.273543 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:49:35.614973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.603072 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Oct 30 21:49:35.813260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.813209 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 21:49:35.844028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.829237 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Oct 30 21:49:35.959507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:35.959463 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 30 21:49:36.053403 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.053365 6808 generic.go:356] "Generic (PLEG): container finished" podID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerID="8cffa7465e40afcbcab708342325b94b9a3710d5bcc8ba8d4ce2775a5e6c545c" exitCode=0 Oct 30 21:49:36.136184 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.120693 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" event={"ID":"bfa354d4-5b05-4fc6-9120-8684f0d2c742","Type":"ContainerDied","Data":"8cffa7465e40afcbcab708342325b94b9a3710d5bcc8ba8d4ce2775a5e6c545c"} Oct 30 21:49:36.171862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.171820 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:36.191095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.191051 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:36.587027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.586343 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Oct 30 21:49:36.678252 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:36.664188 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:49:36.721761 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.719049 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:36.788008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.763544 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 21:49:36.891225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.852034 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:36.980759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:36.980522 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 30 21:49:37.027883 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.027850 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:49:37.182380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.175653 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:37.182380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.177196 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:49:37.309425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.309053 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:49:37.335401 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.335368 6808 generic.go:356] "Generic (PLEG): container finished" podID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerID="145434cc8bc7436f2f98e6f8c210e173a8d638e7be843e61924fe450a13c5c53" exitCode=0 Oct 30 21:49:37.335880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.335848 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" event={"ID":"7ec09683-4e3f-4351-9faf-9cccccd36f9f","Type":"ContainerDied","Data":"145434cc8bc7436f2f98e6f8c210e173a8d638e7be843e61924fe450a13c5c53"} Oct 30 21:49:37.338815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.338790 6808 generic.go:356] "Generic (PLEG): container finished" podID="48ec31e2-b484-45a8-9888-35c505e1752b" containerID="4aacb30c4c88a8ba05bcd50e311af5d703d94bfddbeaf422c374f7916311ce57" exitCode=0 Oct 30 21:49:37.339081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.339062 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" event={"ID":"48ec31e2-b484-45a8-9888-35c505e1752b","Type":"ContainerDied","Data":"4aacb30c4c88a8ba05bcd50e311af5d703d94bfddbeaf422c374f7916311ce57"} Oct 30 21:49:37.477515 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.477152 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:49:37.568078 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.545166 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache\") pod \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " Oct 30 21:49:37.619492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.586881 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache" pod "bfa354d4-5b05-4fc6-9120-8684f0d2c742" (UID: "bfa354d4-5b05-4fc6-9120-8684f0d2c742"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:37.635032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.630203 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Oct 30 21:49:37.635652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.635525 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm\") pod \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " Oct 30 21:49:37.636167 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.636135 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp\" (UniqueName: \"kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp\") pod \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " Oct 30 21:49:37.637008 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.636991 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home\") pod \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " Oct 30 21:49:37.653722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.653687 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs\") pod \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " Oct 30 21:49:37.654442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.654406 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir\") pod \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\" (UID: \"bfa354d4-5b05-4fc6-9120-8684f0d2c742\") " Oct 30 21:49:37.664498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.650493 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home" pod "bfa354d4-5b05-4fc6-9120-8684f0d2c742" (UID: "bfa354d4-5b05-4fc6-9120-8684f0d2c742"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:37.664718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.662512 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp" pod "bfa354d4-5b05-4fc6-9120-8684f0d2c742" (UID: "bfa354d4-5b05-4fc6-9120-8684f0d2c742"). InnerVolumeSpecName "kube-api-access-zwgpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:37.664718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.664024 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir" pod "bfa354d4-5b05-4fc6-9120-8684f0d2c742" (UID: "bfa354d4-5b05-4fc6-9120-8684f0d2c742"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:37.667381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.667360 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:37.667638 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.667621 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwgpp\" (UniqueName: \"kubernetes.io/projected/bfa354d4-5b05-4fc6-9120-8684f0d2c742-kube-api-access-zwgpp\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:37.667854 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.667838 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:37.668132 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.668117 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:37.675492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.675450 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs" pod "bfa354d4-5b05-4fc6-9120-8684f0d2c742" (UID: "bfa354d4-5b05-4fc6-9120-8684f0d2c742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:37.738209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.726492 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:37.738209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.727343 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:49:37.764243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.763198 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 21:49:37.764243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.764067 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:37.779023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.773565 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfa354d4-5b05-4fc6-9120-8684f0d2c742-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:37.788891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.788855 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 21:49:37.828783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.828100 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm" pod "bfa354d4-5b05-4fc6-9120-8684f0d2c742" (UID: "bfa354d4-5b05-4fc6-9120-8684f0d2c742"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:37.906139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:37.887974 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfa354d4-5b05-4fc6-9120-8684f0d2c742-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:38.288343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.264872 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 30 21:49:38.288343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.280883 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:38.781531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.781425 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 21:49:38.844040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.835101 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:49:38.902981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.902919 6808 generic.go:356] "Generic (PLEG): container finished" podID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerID="d601ceed50595c9e926212421e2bf814baac0dab4cf68da822879a2a23e00b64" exitCode=0 Oct 30 21:49:38.903172 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.903101 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" event={"ID":"493dcec4-f99e-4ba5-9db1-87868c28af56","Type":"ContainerDied","Data":"d601ceed50595c9e926212421e2bf814baac0dab4cf68da822879a2a23e00b64"} Oct 30 21:49:38.923009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.909102 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:49:38.962215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:38.962178 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 30 21:49:39.034426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.034331 6808 generic.go:356] "Generic (PLEG): container finished" podID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerID="885ae0f9f0bf71f69fee36fcd5afa1f5cce63fa9b474dbeba4b0fa914380065a" exitCode=0 Oct 30 21:49:39.104973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.041379 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs\") pod \"48ec31e2-b484-45a8-9888-35c505e1752b\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " Oct 30 21:49:39.148974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.071909 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" event={"ID":"f46ac04d-5ae1-43f1-be29-5a08872cb890","Type":"ContainerDied","Data":"885ae0f9f0bf71f69fee36fcd5afa1f5cce63fa9b474dbeba4b0fa914380065a"} Oct 30 21:49:39.150283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.150252 6808 scope.go:122] "RemoveContainer" containerID="8c370170f2eb312ffca0ef4482d9fa8342e81e5965cc3cf17c56935b8ead6714" Oct 30 21:49:39.150678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.150651 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:39.152278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.152256 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir\") pod \"48ec31e2-b484-45a8-9888-35c505e1752b\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " Oct 30 21:49:39.152672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.152647 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67\" (UniqueName: \"kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67\") pod \"48ec31e2-b484-45a8-9888-35c505e1752b\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " Oct 30 21:49:39.152853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.152833 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache\") pod \"48ec31e2-b484-45a8-9888-35c505e1752b\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " Oct 30 21:49:39.152990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.152974 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home\") pod \"48ec31e2-b484-45a8-9888-35c505e1752b\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " Oct 30 21:49:39.170973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.153207 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm\") pod \"48ec31e2-b484-45a8-9888-35c505e1752b\" (UID: \"48ec31e2-b484-45a8-9888-35c505e1752b\") " Oct 30 21:49:39.177885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.176437 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir" pod "48ec31e2-b484-45a8-9888-35c505e1752b" (UID: "48ec31e2-b484-45a8-9888-35c505e1752b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:39.192800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.192761 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache" pod "48ec31e2-b484-45a8-9888-35c505e1752b" (UID: "48ec31e2-b484-45a8-9888-35c505e1752b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:39.193136 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.193022 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home" pod "48ec31e2-b484-45a8-9888-35c505e1752b" (UID: "48ec31e2-b484-45a8-9888-35c505e1752b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:39.217626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.217600 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" Oct 30 21:49:39.221247 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.221186 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm" event={"ID":"bfa354d4-5b05-4fc6-9120-8684f0d2c742","Type":"ContainerDied","Data":"fb8c8e382930277a483cd35678950c6c308dcf1e46c69348559a3a4cbeb87a4c"} Oct 30 21:49:39.271285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.270783 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs" pod "48ec31e2-b484-45a8-9888-35c505e1752b" (UID: "48ec31e2-b484-45a8-9888-35c505e1752b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:39.336691 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.336645 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec31e2-b484-45a8-9888-35c505e1752b-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.418380 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.418351 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.418651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.418636 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.418799 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.418785 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.418919 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.364001 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:49:39.550072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.549930 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:49:39.563987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.553457 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:49:39.614077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.609023 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" containerID="cri-o://a353c91f81eb6ba23c994573bad78fa51e9c238b720b98085d6c7bc3b60704d5" gracePeriod=45 Oct 30 21:49:39.617199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.615527 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:49:39.617199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.615896 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 21:49:39.625613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.625579 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm" pod "48ec31e2-b484-45a8-9888-35c505e1752b" (UID: "48ec31e2-b484-45a8-9888-35c505e1752b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:39.627671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.627643 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67" pod "48ec31e2-b484-45a8-9888-35c505e1752b" (UID: "48ec31e2-b484-45a8-9888-35c505e1752b"). InnerVolumeSpecName "kube-api-access-6pl67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:39.635209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.635174 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pl67\" (UniqueName: \"kubernetes.io/projected/48ec31e2-b484-45a8-9888-35c505e1752b-kube-api-access-6pl67\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.635435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.635417 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48ec31e2-b484-45a8-9888-35c505e1752b-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.638636 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.638608 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": read tcp 10.134.0.2:44170->10.134.0.112:8000: read: connection reset by peer" Oct 30 21:49:39.638855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.638838 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:49:39.852976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.804230 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 21:49:39.960297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:39.919176 6808 scope.go:122] "RemoveContainer" containerID="8cffa7465e40afcbcab708342325b94b9a3710d5bcc8ba8d4ce2775a5e6c545c" Oct 30 21:49:40.076010 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.068008 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:49:40.177126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.144964 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": dial tcp 10.134.0.112:8000: connect: connection refused" Oct 30 21:49:40.215624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.214345 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:49:40.299123 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.299093 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:49:40.299416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.299393 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm"] Oct 30 21:49:40.308962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.308919 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-2-w0-000f64-kserve-57b5476d6-8d2zm"] Oct 30 21:49:40.403526 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.403490 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" Oct 30 21:49:40.418748 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.418441 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd" event={"ID":"48ec31e2-b484-45a8-9888-35c505e1752b","Type":"ContainerDied","Data":"bea616915ea8904c9c64832d6a95b9e0adea077798459959e97e7dcb36e26b14"} Oct 30 21:49:40.433811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.419193 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:49:40.433811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.419695 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:49:40.478161 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.478110 6808 scope.go:122] "RemoveContainer" containerID="4aacb30c4c88a8ba05bcd50e311af5d703d94bfddbeaf422c374f7916311ce57" Oct 30 21:49:40.632165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.632124 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:49:40.637721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.632904 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:49:40.709149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.672706 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:49:40.807211 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.801441 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.877453 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c"] Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.877530 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6"] Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878589 6808 cpu_manager.go:414] "containerMap: removing container" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878610 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878667 6808 cpu_manager.go:414] "containerMap: removing container" podUID="0201a121-f31a-41b4-9e87-3edd447c1e4a" containerName="payload-processing" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878676 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0201a121-f31a-41b4-9e87-3edd447c1e4a" containerName="payload-processing" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878696 6808 cpu_manager.go:414] "containerMap: removing container" podUID="cee00dab-e07c-4bb0-9b5a-7647833675d0" containerName="maas-api" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878704 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee00dab-e07c-4bb0-9b5a-7647833675d0" containerName="maas-api" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878724 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878730 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878962 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="0201a121-f31a-41b4-9e87-3edd447c1e4a" containerName="payload-processing" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878977 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee00dab-e07c-4bb0-9b5a-7647833675d0" containerName="maas-api" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.878988 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" containerName="main" Oct 30 21:49:40.885143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.879007 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" containerName="main" Oct 30 21:49:40.956971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.896871 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" containerID="cri-o://220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d" gracePeriod=30 Oct 30 21:49:40.984309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.972149 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:40.991109 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:40.988818 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": dial tcp 10.134.0.112:8000: connect: connection refused" Oct 30 21:49:41.088073 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.080155 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:49:41.160026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.098405 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.231308 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql"] Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.232420 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.232440 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.235509 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.235563 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.235636 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.235817 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.235897 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.235970 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.236039 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.236353 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.242849 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.236418 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw\") pod \"f46ac04d-5ae1-43f1-be29-5a08872cb890\" (UID: \"f46ac04d-5ae1-43f1-be29-5a08872cb890\") " Oct 30 21:49:41.266356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.256156 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:41.266356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.248046 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:41.266356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.260172 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:41.266356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.265162 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:41.266356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.265517 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "kube-api-access-q5vhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:41.267398 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.267354 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.268512 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.268494 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" Oct 30 21:49:41.272627 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.272603 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:41.326576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.314162 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:49:41.340024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.314329 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6"] Oct 30 21:49:41.370110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.370081 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql"] Oct 30 21:49:41.394577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.394551 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:49:41.418976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.394806 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:49:41.583075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.583039 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6"] Oct 30 21:49:41.583355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.314412 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:41.584180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.333339 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:41.605911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.394206 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:41.606146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.500149 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo" pod "f46ac04d-5ae1-43f1-be29-5a08872cb890" (UID: "f46ac04d-5ae1-43f1-be29-5a08872cb890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:41.606273 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.509120 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.606509 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.606488 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.663203 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.663166 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.686272 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.686341 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.686514 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrbb\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.686598 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.686628 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.686942 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:41.729471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.663018 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql"] Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:41.795757 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[credential-socket istio-data istio-envoy istio-podinfo istio-token istiod-ca-cert kube-api-access-jrrbb workload-certs workload-socket], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" podUID="86e4ead6-cb7c-44a1-83a1-169cfdb4effc" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795862 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795881 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795908 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795924 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795941 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795976 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vhw\" (UniqueName: \"kubernetes.io/projected/f46ac04d-5ae1-43f1-be29-5a08872cb890-kube-api-access-q5vhw\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.795996 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.796011 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ac04d-5ae1-43f1-be29-5a08872cb890-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.796026 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ac04d-5ae1-43f1-be29-5a08872cb890-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.815761 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:49:41.981233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.834786 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:49:41.992378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.335899 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:49:41.992378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:41.986632 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 21:49:42.023173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.023129 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.041492 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm\") pod \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.041559 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home\") pod \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.041690 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache\") pod \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.041746 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir\") pod \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.041990 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs\") pod \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.042114 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj\" (UniqueName: \"kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj\") pod \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\" (UID: \"7ec09683-4e3f-4351-9faf-9cccccd36f9f\") " Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043453 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043540 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043677 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043717 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4qj\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043751 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043795 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043839 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.045006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.043925 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.109221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.102809 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm" pod "7ec09683-4e3f-4351-9faf-9cccccd36f9f" (UID: "7ec09683-4e3f-4351-9faf-9cccccd36f9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:42.109221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.103212 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home" pod "7ec09683-4e3f-4351-9faf-9cccccd36f9f" (UID: "7ec09683-4e3f-4351-9faf-9cccccd36f9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:42.109221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.103414 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache" pod "7ec09683-4e3f-4351-9faf-9cccccd36f9f" (UID: "7ec09683-4e3f-4351-9faf-9cccccd36f9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:42.109221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.103579 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir" pod "7ec09683-4e3f-4351-9faf-9cccccd36f9f" (UID: "7ec09683-4e3f-4351-9faf-9cccccd36f9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:42.109474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.109359 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:42.111310 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.111284 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.187456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.187081 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.187456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.187164 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.187456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.187223 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.187456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.187335 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.188213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.187882 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrbb\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.188213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.188023 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.188213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.188075 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.188213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.188154 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.226449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.202045 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 21:49:42.227445 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.142708 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:42.316544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.282781 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf"] Oct 30 21:49:42.316544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.283205 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" containerID="cri-o://8af2002b72a3a54762ed329dec53fff462ff387fbb9b04a2efcab376c00cb565" gracePeriod=30 Oct 30 21:49:42.316544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.285091 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:42.329628 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.329170 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.342028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.337455 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:42.395163 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:42.363221 6808 projected.go:196] Error preparing data for projected volume kube-api-access-jrrbb for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:42.395163 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:42.363312 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb podName:86e4ead6-cb7c-44a1-83a1-169cfdb4effc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:42.863289346 +0000 UTC m=+1245.058980464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jrrbb" (UniqueName: "kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:42.398855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.395568 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 21:49:42.398855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.398337 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.398855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.398543 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.438399 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.438364 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.765001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.763326 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs" pod "7ec09683-4e3f-4351-9faf-9cccccd36f9f" (UID: "7ec09683-4e3f-4351-9faf-9cccccd36f9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:42.765001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.763555 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:42.801179 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:42.782743 6808 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[credential-socket istio-data istio-envoy istio-podinfo istio-token istiod-ca-cert kube-api-access-br4qj workload-certs workload-socket], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" podUID="35169fc5-ece2-407e-bf6a-5fcfb754f5f6" Oct 30 21:49:42.801179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.782828 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:42.801179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.782860 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:42.801179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.782874 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec09683-4e3f-4351-9faf-9cccccd36f9f-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:42.810645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.810618 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" Oct 30 21:49:42.812093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.811089 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.812287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.811410 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:42.855641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.823811 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:42.855641 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:42.830814 6808 projected.go:196] Error preparing data for projected volume istio-token for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:42.855641 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:42.830891 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token podName:86e4ead6-cb7c-44a1-83a1-169cfdb4effc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:43.330869547 +0000 UTC m=+1245.526560649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-token" (UniqueName: "kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:42.855641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.834549 6808 generic.go:356] "Generic (PLEG): container finished" podID="3c551711-d8a4-4824-aee5-d958286e738d" containerID="a353c91f81eb6ba23c994573bad78fa51e9c238b720b98085d6c7bc3b60704d5" exitCode=0 Oct 30 21:49:42.855641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.839388 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:49:42.858833 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.857828 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" Oct 30 21:49:42.925243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.922705 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj" pod "7ec09683-4e3f-4351-9faf-9cccccd36f9f" (UID: "7ec09683-4e3f-4351-9faf-9cccccd36f9f"). InnerVolumeSpecName "kube-api-access-xmxnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:42.926120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.926089 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache\") pod \"3c551711-d8a4-4824-aee5-d958286e738d\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " Oct 30 21:49:42.926341 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.926325 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir\") pod \"493dcec4-f99e-4ba5-9db1-87868c28af56\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " Oct 30 21:49:42.926511 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.926496 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh\" (UniqueName: \"kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh\") pod \"493dcec4-f99e-4ba5-9db1-87868c28af56\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " Oct 30 21:49:42.926673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.926657 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq\" (UniqueName: \"kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq\") pod \"3c551711-d8a4-4824-aee5-d958286e738d\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " Oct 30 21:49:42.926858 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.926841 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm\") pod \"3c551711-d8a4-4824-aee5-d958286e738d\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " Oct 30 21:49:42.927116 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.927098 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm\") pod \"493dcec4-f99e-4ba5-9db1-87868c28af56\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " Oct 30 21:49:42.927264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.927249 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home\") pod \"493dcec4-f99e-4ba5-9db1-87868c28af56\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " Oct 30 21:49:42.927383 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.927368 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs\" (UniqueName: \"kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs\") pod \"493dcec4-f99e-4ba5-9db1-87868c28af56\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " Oct 30 21:49:42.961575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.927488 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs\") pod \"3c551711-d8a4-4824-aee5-d958286e738d\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " Oct 30 21:49:43.097060 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.097026 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir\") pod \"3c551711-d8a4-4824-aee5-d958286e738d\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " Oct 30 21:49:43.110178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.110148 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home\") pod \"3c551711-d8a4-4824-aee5-d958286e738d\" (UID: \"3c551711-d8a4-4824-aee5-d958286e738d\") " Oct 30 21:49:43.110502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.110482 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache\") pod \"493dcec4-f99e-4ba5-9db1-87868c28af56\" (UID: \"493dcec4-f99e-4ba5-9db1-87868c28af56\") " Oct 30 21:49:43.153283 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.111307 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.171757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.945574 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs" pod "3c551711-d8a4-4824-aee5-d958286e738d" (UID: "3c551711-d8a4-4824-aee5-d958286e738d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:43.178848 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.017532 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded" Oct 30 21:49:43.220110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.018809 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" Oct 30 21:49:43.228406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.069653 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.261325 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.972694 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir" pod "493dcec4-f99e-4ba5-9db1-87868c28af56" (UID: "493dcec4-f99e-4ba5-9db1-87868c28af56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.972816 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh" pod "493dcec4-f99e-4ba5-9db1-87868c28af56" (UID: "493dcec4-f99e-4ba5-9db1-87868c28af56"). InnerVolumeSpecName "kube-api-access-lzzfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:42.981122 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache" pod "3c551711-d8a4-4824-aee5-d958286e738d" (UID: "3c551711-d8a4-4824-aee5-d958286e738d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.016182 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home" pod "493dcec4-f99e-4ba5-9db1-87868c28af56" (UID: "493dcec4-f99e-4ba5-9db1-87868c28af56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.039603 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq" pod "3c551711-d8a4-4824-aee5-d958286e738d" (UID: "3c551711-d8a4-4824-aee5-d958286e738d"). InnerVolumeSpecName "kube-api-access-dk2mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.068299 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm" pod "3c551711-d8a4-4824-aee5-d958286e738d" (UID: "3c551711-d8a4-4824-aee5-d958286e738d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.068363 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm" pod "493dcec4-f99e-4ba5-9db1-87868c28af56" (UID: "493dcec4-f99e-4ba5-9db1-87868c28af56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.069062 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs" pod "493dcec4-f99e-4ba5-9db1-87868c28af56" (UID: "493dcec4-f99e-4ba5-9db1-87868c28af56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.111192 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home" pod "3c551711-d8a4-4824-aee5-d958286e738d" (UID: "3c551711-d8a4-4824-aee5-d958286e738d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.111350 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir" pod "3c551711-d8a4-4824-aee5-d958286e738d" (UID: "3c551711-d8a4-4824-aee5-d958286e738d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.277110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.124140 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache" pod "493dcec4-f99e-4ba5-9db1-87868c28af56" (UID: "493dcec4-f99e-4ba5-9db1-87868c28af56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:43.297990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.289937 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.297990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.290031 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.424975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.398824 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:43.424975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.413536 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.459530 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.459657 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrbb\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.459742 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.459813 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.459904 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.459989 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460182 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4qj\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460218 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460257 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460435 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460450 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460463 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493dcec4-f99e-4ba5-9db1-87868c28af56-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460476 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551711-d8a4-4824-aee5-d958286e738d-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460490 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460503 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460516 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec09683-4e3f-4351-9faf-9cccccd36f9f-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460528 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460543 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxnj\" (UniqueName: \"kubernetes.io/projected/7ec09683-4e3f-4351-9faf-9cccccd36f9f-kube-api-access-xmxnj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460558 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460590 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493dcec4-f99e-4ba5-9db1-87868c28af56-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460607 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzfh\" (UniqueName: \"kubernetes.io/projected/493dcec4-f99e-4ba5-9db1-87868c28af56-kube-api-access-lzzfh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460622 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2mq\" (UniqueName: \"kubernetes.io/projected/3c551711-d8a4-4824-aee5-d958286e738d-kube-api-access-dk2mq\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.462143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.460637 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c551711-d8a4-4824-aee5-d958286e738d-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:43.469993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.463771 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.486974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.473077 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.561685 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.489257 6808 projected.go:196] Error preparing data for projected volume kube-api-access-jrrbb for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.563053 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.519354 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.567049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.520327 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.567221 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.489287 6808 projected.go:196] Error preparing data for projected volume istio-token for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.571044 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.536496 6808 projected.go:196] Error preparing data for projected volume kube-api-access-br4qj for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.571204 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.536535 6808 projected.go:196] Error preparing data for projected volume istio-token for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.571310 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.569032 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb podName:86e4ead6-cb7c-44a1-83a1-169cfdb4effc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:44.568995092 +0000 UTC m=+1246.764686195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrrbb" (UniqueName: "kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.609179 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.571353 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:43.609416 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.609231 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token podName:35169fc5-ece2-407e-bf6a-5fcfb754f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:44.109195704 +0000 UTC m=+1246.304886814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-token" (UniqueName: "kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.621194 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.609557 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj podName:35169fc5-ece2-407e-bf6a-5fcfb754f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:44.109529641 +0000 UTC m=+1246.305220755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-br4qj" (UniqueName: "kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.622149 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.622124 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token podName:86e4ead6-cb7c-44a1-83a1-169cfdb4effc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:44.622086574 +0000 UTC m=+1246.817777684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "istio-token" (UniqueName: "kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:43.749145 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.749046 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:43.795104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.787422 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa354d4-5b05-4fc6-9120-8684f0d2c742" path="/var/lib/kubelet/pods/bfa354d4-5b05-4fc6-9120-8684f0d2c742/volumes" Oct 30 21:49:43.884833 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:43.884799 6808 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.199s" Oct 30 21:49:43.907972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.885143 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:49:43.908244 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.908218 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k" event={"ID":"7ec09683-4e3f-4351-9faf-9cccccd36f9f","Type":"ContainerDied","Data":"bf5d1336ffc1f27680f22f58fb17aa37e8dc8e54771838cc03ff5591af859e6f"} Oct 30 21:49:43.927997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.908515 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" event={"ID":"3c551711-d8a4-4824-aee5-d958286e738d","Type":"ContainerDied","Data":"a353c91f81eb6ba23c994573bad78fa51e9c238b720b98085d6c7bc3b60704d5"} Oct 30 21:49:43.928284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.928247 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q" event={"ID":"493dcec4-f99e-4ba5-9db1-87868c28af56","Type":"ContainerDied","Data":"eb5613498d15eb2508b7576d310f7d941de9828c8411b3981d990212442a9645"} Oct 30 21:49:43.929743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.907577 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:43.930588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.909750 6808 scope.go:122] "RemoveContainer" containerID="145434cc8bc7436f2f98e6f8c210e173a8d638e7be843e61924fe450a13c5c53" Oct 30 21:49:43.978944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.978914 6808 scope.go:122] "RemoveContainer" containerID="a353c91f81eb6ba23c994573bad78fa51e9c238b720b98085d6c7bc3b60704d5" Oct 30 21:49:43.996721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.996684 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq" event={"ID":"f46ac04d-5ae1-43f1-be29-5a08872cb890","Type":"ContainerDied","Data":"1bb237dc55bcd27d36eac21caf5b918466f7d8c06121f1bd1d89218a55aeed2c"} Oct 30 21:49:43.996976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.996938 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd"] Oct 30 21:49:43.997289 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:43.997269 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/simulator-w0-000f64-kserve-8695898b47-rrqmd"] Oct 30 21:49:44.046174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.045316 6808 scope.go:122] "RemoveContainer" containerID="d601ceed50595c9e926212421e2bf814baac0dab4cf68da822879a2a23e00b64" Oct 30 21:49:44.174066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.115603 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4qj\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:44.174066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.115811 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:44.187926 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.187443 6808 projected.go:196] Error preparing data for projected volume kube-api-access-br4qj for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.187926 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.187518 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj podName:35169fc5-ece2-407e-bf6a-5fcfb754f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.187496051 +0000 UTC m=+1247.383187161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-br4qj" (UniqueName: "kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.187926 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.187841 6808 projected.go:196] Error preparing data for projected volume istio-token for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.187926 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.187888 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token podName:35169fc5-ece2-407e-bf6a-5fcfb754f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.187874368 +0000 UTC m=+1247.383565480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "istio-token" (UniqueName: "kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.238980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.238066 6808 scope.go:122] "RemoveContainer" containerID="885ae0f9f0bf71f69fee36fcd5afa1f5cce63fa9b474dbeba4b0fa914380065a" Oct 30 21:49:44.311275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.275026 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:44.311275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.299828 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:49:44.311275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.309400 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" Oct 30 21:49:44.317965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.317703 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w" event={"ID":"3c551711-d8a4-4824-aee5-d958286e738d","Type":"ContainerDied","Data":"15a6408b2435018ab1b65ac949890b903feee6c8af4905215b3f4b715bf3ef67"} Oct 30 21:49:44.318249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.318229 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:44.318859 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.318839 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:44.322007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.320100 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:49:44.330993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.327604 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k"] Oct 30 21:49:44.331168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.331143 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/premium-w0-000f64-kserve-5758cf7c78-6dw8k"] Oct 30 21:49:44.342038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.342013 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq"] Oct 30 21:49:44.357050 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.357021 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd3c4c-openshift-default-6c84485d6f-cxrbq"] Oct 30 21:49:44.406014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.405210 6808 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:44.406014 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.405261 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:44.407021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.406997 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:49:44.414352 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.414330 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q"] Oct 30 21:49:44.423656 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.423630 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-000f64/distinct-w0-000f64-kserve-6744b7b5df-ld78q"] Oct 30 21:49:44.473208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.463689 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:44.474998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.473341 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:44.532300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.532270 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.534124 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.532670 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.534421 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.534405 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.535104 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.535078 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.535322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.534735 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:44.535718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.535604 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.535837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.535293 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.535837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.535781 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.536000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.535977 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.536110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536084 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.536264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536241 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.536416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536295 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.536416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536363 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.536654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536426 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.536654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536534 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.536835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536663 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.536835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536709 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.536835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536789 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert\") pod \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " Oct 30 21:49:44.537221 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.536850 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket\") pod \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " Oct 30 21:49:44.537594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.537570 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.538276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.537717 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.538276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.537942 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.538276 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.538242 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.538347 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.538562 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539023 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539045 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539061 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539075 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539088 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539102 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539115 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539128 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539142 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539155 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.539416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.539168 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.578381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.578339 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:44.598025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.597982 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo" pod "86e4ead6-cb7c-44a1-83a1-169cfdb4effc" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:44.598249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.598227 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w"] Oct 30 21:49:44.598671 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.598642 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo" pod "35169fc5-ece2-407e-bf6a-5fcfb754f5f6" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:44.603825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.603798 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-08aafeac/e2e-lifecycle-model-08aafeac-kserve-fb9547b9c-jqv7w"] Oct 30 21:49:44.641015 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.640835 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrbb\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:44.641199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.641056 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6\" (UID: \"86e4ead6-cb7c-44a1-83a1-169cfdb4effc\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:44.641199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.641153 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.641199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.641171 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.641199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:44.641187 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.651879 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.651852 6808 projected.go:196] Error preparing data for projected volume kube-api-access-jrrbb for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.652018 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.651926 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb podName:86e4ead6-cb7c-44a1-83a1-169cfdb4effc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:46.651906432 +0000 UTC m=+1248.847597544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrrbb" (UniqueName: "kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.652298 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.652282 6808 projected.go:196] Error preparing data for projected volume istio-token for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:44.652359 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:44.652348 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token podName:86e4ead6-cb7c-44a1-83a1-169cfdb4effc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:46.652333652 +0000 UTC m=+1248.848024763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "istio-token" (UniqueName: "kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" (UID: "86e4ead6-cb7c-44a1-83a1-169cfdb4effc") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:45.290938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.290887 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4qj\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:45.291685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.291658 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql\" (UID: \"35169fc5-ece2-407e-bf6a-5fcfb754f5f6\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:45.295138 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:45.295115 6808 projected.go:196] Error preparing data for projected volume kube-api-access-br4qj for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:45.295238 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:45.295187 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj podName:35169fc5-ece2-407e-bf6a-5fcfb754f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:47.295166774 +0000 UTC m=+1249.490857879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-br4qj" (UniqueName: "kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:45.295580 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:45.295564 6808 projected.go:196] Error preparing data for projected volume istio-token for pod openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql: failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:45.295640 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:45.295619 6808 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token podName:35169fc5-ece2-407e-bf6a-5fcfb754f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:47.295603279 +0000 UTC m=+1249.491294379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "istio-token" (UniqueName: "kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token") pod "e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" (UID: "35169fc5-ece2-407e-bf6a-5fcfb754f5f6") : failed to fetch token: serviceaccounts "e2e-worker-w6-7dfff0-openshift-default" not found Oct 30 21:49:45.322853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.322823 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6" Oct 30 21:49:45.323056 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.322832 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql" Oct 30 21:49:45.385825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.385766 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6"] Oct 30 21:49:45.390805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.390776 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-877b6"] Oct 30 21:49:45.477761 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.477714 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql"] Oct 30 21:49:45.482714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.482682 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-f55ql"] Oct 30 21:49:45.500997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.495815 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrbb\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-kube-api-access-jrrbb\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:45.500997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.495847 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e4ead6-cb7c-44a1-83a1-169cfdb4effc-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:45.598359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.598279 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:45.598359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.598316 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br4qj\" (UniqueName: \"kubernetes.io/projected/35169fc5-ece2-407e-bf6a-5fcfb754f5f6-kube-api-access-br4qj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:45.974453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.969140 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35169fc5-ece2-407e-bf6a-5fcfb754f5f6" path="/var/lib/kubelet/pods/35169fc5-ece2-407e-bf6a-5fcfb754f5f6/volumes" Oct 30 21:49:45.975409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.975378 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c551711-d8a4-4824-aee5-d958286e738d" path="/var/lib/kubelet/pods/3c551711-d8a4-4824-aee5-d958286e738d/volumes" Oct 30 21:49:45.976298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.976275 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ec31e2-b484-45a8-9888-35c505e1752b" path="/var/lib/kubelet/pods/48ec31e2-b484-45a8-9888-35c505e1752b/volumes" Oct 30 21:49:45.982690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.982660 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" path="/var/lib/kubelet/pods/493dcec4-f99e-4ba5-9db1-87868c28af56/volumes" Oct 30 21:49:45.983558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.983537 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" path="/var/lib/kubelet/pods/7ec09683-4e3f-4351-9faf-9cccccd36f9f/volumes" Oct 30 21:49:45.984486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.984465 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e4ead6-cb7c-44a1-83a1-169cfdb4effc" path="/var/lib/kubelet/pods/86e4ead6-cb7c-44a1-83a1-169cfdb4effc/volumes" Oct 30 21:49:45.991806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:45.991768 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" path="/var/lib/kubelet/pods/f46ac04d-5ae1-43f1-be29-5a08872cb890/volumes" Oct 30 21:49:46.620996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.618365 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:49:46.726887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.726745 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.726887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.726795 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.726887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.726833 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.727200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.726918 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.727200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.727082 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.727200 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.727113 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.727385 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.727436 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.727486 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket\") pod \"a6e78663-fd4e-451e-a3f4-380804008a8a\" (UID: \"a6e78663-fd4e-451e-a3f4-380804008a8a\") " Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.727869 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.728126 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.728665 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.728673 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.728690 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e78663-fd4e-451e-a3f4-380804008a8a-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.728806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.728769 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.729381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.728968 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.731520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.731480 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.731775 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.731748 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:46.734086 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.734057 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.734210 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.734159 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb" pod "a6e78663-fd4e-451e-a3f4-380804008a8a" (UID: "a6e78663-fd4e-451e-a3f4-380804008a8a"). InnerVolumeSpecName "kube-api-access-66fqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899548 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fqb\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-kube-api-access-66fqb\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899582 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899597 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899610 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899624 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899638 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.903307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:46.899651 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e78663-fd4e-451e-a3f4-380804008a8a-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.347705 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.347613 6808 generic.go:356] "Generic (PLEG): container finished" podID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerID="220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d" exitCode=0 Oct 30 21:49:47.347879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.347773 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" event={"ID":"a6e78663-fd4e-451e-a3f4-380804008a8a","Type":"ContainerDied","Data":"220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d"} Oct 30 21:49:47.347879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.347860 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" event={"ID":"a6e78663-fd4e-451e-a3f4-380804008a8a","Type":"ContainerDied","Data":"f04e5cc2c724f262251d57a97b8cb5b125e9fe2e8fdd05277e711423f6b99891"} Oct 30 21:49:47.348059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.347888 6808 scope.go:122] "RemoveContainer" containerID="220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d" Oct 30 21:49:47.348118 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.348079 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c" Oct 30 21:49:47.363625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.363598 6808 scope.go:122] "RemoveContainer" containerID="220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d" Oct 30 21:49:47.363999 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:47.363967 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d\": container with ID starting with 220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d not found: ID does not exist" containerID="220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d" Oct 30 21:49:47.364109 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.364009 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d"} err="failed to get container status \"220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d\": rpc error: code = NotFound desc = could not find container \"220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d\": container with ID starting with 220872d458de599066b421b5e95746b0af87da3125732cf848c08f1398f3f81d not found: ID does not exist" Oct 30 21:49:47.381411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.381382 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c"] Oct 30 21:49:47.391119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:47.391095 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-d726c"] Oct 30 21:49:48.037047 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.037006 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" path="/var/lib/kubelet/pods/a6e78663-fd4e-451e-a3f4-380804008a8a/volumes" Oct 30 21:49:48.375418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.373721 6808 generic.go:356] "Generic (PLEG): container finished" podID="79d92887-ea5d-4ced-880e-de31ce124c20" containerID="8af2002b72a3a54762ed329dec53fff462ff387fbb9b04a2efcab376c00cb565" exitCode=0 Oct 30 21:49:48.375418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.374009 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" event={"ID":"79d92887-ea5d-4ced-880e-de31ce124c20","Type":"ContainerDied","Data":"8af2002b72a3a54762ed329dec53fff462ff387fbb9b04a2efcab376c00cb565"} Oct 30 21:49:48.637362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.637333 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:49:48.713997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.713957 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714113 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714155 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714237 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714275 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714410 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714331 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714453 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714572 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:48.714688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714587 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.714757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714732 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token\") pod \"79d92887-ea5d-4ced-880e-de31ce124c20\" (UID: \"79d92887-ea5d-4ced-880e-de31ce124c20\") " Oct 30 21:49:48.715156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714909 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:48.715156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.714935 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:48.715156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.715012 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:48.715540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.715515 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d92887-ea5d-4ced-880e-de31ce124c20-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.715614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.715547 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.715614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.715563 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.715614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.715577 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.715777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.715640 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:48.718961 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.717462 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:48.777905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.747420 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:48.777905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.747547 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:48.777905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.750123 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l" pod "79d92887-ea5d-4ced-880e-de31ce124c20" (UID: "79d92887-ea5d-4ced-880e-de31ce124c20"). InnerVolumeSpecName "kube-api-access-jkc6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:48.777905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.755278 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx"] Oct 30 21:49:48.777905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.756179 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" containerID="cri-o://a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935" gracePeriod=30 Oct 30 21:49:48.817386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.817341 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.817386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.817379 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.817639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.817399 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkc6l\" (UniqueName: \"kubernetes.io/projected/79d92887-ea5d-4ced-880e-de31ce124c20-kube-api-access-jkc6l\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.817639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.817417 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d92887-ea5d-4ced-880e-de31ce124c20-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:48.817639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:48.817430 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d92887-ea5d-4ced-880e-de31ce124c20-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.387750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:49.387719 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" Oct 30 21:49:49.387750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:49.387732 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf" event={"ID":"79d92887-ea5d-4ced-880e-de31ce124c20","Type":"ContainerDied","Data":"eaa7baf866ca1a3c50a13f80d06660540d79a59c9de577c89f8804f4cb66c8d1"} Oct 30 21:49:49.388335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:49.387815 6808 scope.go:122] "RemoveContainer" containerID="8af2002b72a3a54762ed329dec53fff462ff387fbb9b04a2efcab376c00cb565" Oct 30 21:49:49.450406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:49.450336 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf"] Oct 30 21:49:49.464477 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:49.464447 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-9rwgf"] Oct 30 21:49:49.936026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:49.935993 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" path="/var/lib/kubelet/pods/79d92887-ea5d-4ced-880e-de31ce124c20/volumes" Oct 30 21:49:50.396516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.396481 6808 generic.go:356] "Generic (PLEG): container finished" podID="5e659ef2-1952-4763-819a-f0034d970c84" containerID="883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be" exitCode=0 Oct 30 21:49:50.397015 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.396557 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" event={"ID":"5e659ef2-1952-4763-819a-f0034d970c84","Type":"ContainerDied","Data":"883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be"} Oct 30 21:49:50.397095 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.397035 6808 scope.go:122] "RemoveContainer" containerID="883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be" Oct 30 21:49:50.398689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.398663 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:50.398752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.398727 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:50.398800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.398763 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:50.613391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:50.613360 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:49:51.406891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:51.406857 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" event={"ID":"5e659ef2-1952-4763-819a-f0034d970c84","Type":"ContainerStarted","Data":"b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b"} Oct 30 21:49:52.062358 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:52.062328 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:49:52.062844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:52.062820 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:52.063065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:52.063039 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:53.065973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:53.058297 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:53.065973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:53.058356 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:54.080975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.079133 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:54.080975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.079211 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:54.213240 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.213148 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:49:54.366062 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.325602 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.398286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.398163 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.398721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.398700 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.399027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.399003 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.407973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.405263 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.410215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.410181 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.424089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.411225 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:54.424323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.421075 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:54.424437 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.421337 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:54.424588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.424569 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.448468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.448406 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:54.463864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.463817 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.464285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.464261 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket\") pod \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\" (UID: \"333f1d92-0c31-4a66-8efb-3fe2c2abd71d\") " Oct 30 21:49:54.465516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.465480 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.465672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.465657 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.465800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.465786 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.465921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.465907 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.466328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.466306 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:54.502189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.501088 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "kube-api-access-cql5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:54.502786 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.502761 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:54.504493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.504468 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:54.504601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.504542 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy" pod "333f1d92-0c31-4a66-8efb-3fe2c2abd71d" (UID: "333f1d92-0c31-4a66-8efb-3fe2c2abd71d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:54.508961 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.508917 6808 generic.go:356] "Generic (PLEG): container finished" podID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerID="a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935" exitCode=0 Oct 30 21:49:54.509067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.508986 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" Oct 30 21:49:54.509067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.508986 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" event={"ID":"333f1d92-0c31-4a66-8efb-3fe2c2abd71d","Type":"ContainerDied","Data":"a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935"} Oct 30 21:49:54.509067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.509040 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx" event={"ID":"333f1d92-0c31-4a66-8efb-3fe2c2abd71d","Type":"ContainerDied","Data":"1f450b9f667b12e259225c4c753d9aac282574af7efb8af64a9ce19333fe0aae"} Oct 30 21:49:54.509067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.509066 6808 scope.go:122] "RemoveContainer" containerID="a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935" Oct 30 21:49:54.523035 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.523016 6808 scope.go:122] "RemoveContainer" containerID="9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa" Oct 30 21:49:54.536640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.536620 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx"] Oct 30 21:49:54.537049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.537029 6808 scope.go:122] "RemoveContainer" containerID="a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935" Oct 30 21:49:54.537464 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:54.537442 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935\": container with ID starting with a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935 not found: ID does not exist" containerID="a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935" Oct 30 21:49:54.537562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.537470 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935"} err="failed to get container status \"a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935\": rpc error: code = NotFound desc = could not find container \"a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935\": container with ID starting with a31719ecae1d3685b8d009477bbed49f7e02dfccbe0f05028fbffa9928e8b935 not found: ID does not exist" Oct 30 21:49:54.537562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.537486 6808 scope.go:122] "RemoveContainer" containerID="9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa" Oct 30 21:49:54.537715 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:54.537698 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa\": container with ID starting with 9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa not found: ID does not exist" containerID="9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa" Oct 30 21:49:54.537791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.537719 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa"} err="failed to get container status \"9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa\": rpc error: code = NotFound desc = could not find container \"9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa\": container with ID starting with 9386101513c33c43a227f6e5d262107b6920aa83056470b3061fcfaa41bcf6aa not found: ID does not exist" Oct 30 21:49:54.556987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.543965 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dd5bf8-openshift-default-5b9d9cf875-s92wx"] Oct 30 21:49:54.576615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.576473 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.576615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.576513 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cql5p\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-kube-api-access-cql5p\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.576615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.576533 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.576615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.576547 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.576615 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.576563 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333f1d92-0c31-4a66-8efb-3fe2c2abd71d-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.616834 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:49:54.616741 6808 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333f1d92_0c31_4a66_8efb_3fe2c2abd71d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:49:54.911172 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.911129 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x"] Oct 30 21:49:54.911429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:54.911387 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" containerID="cri-o://cd7b5c2b769c4bb2c7d4edc893857d0d7a5bd81e96722154e09174253b18e7e3" gracePeriod=30 Oct 30 21:49:55.062101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:55.054701 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:55.062101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:55.054760 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:56.122970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.122713 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" path="/var/lib/kubelet/pods/333f1d92-0c31-4a66-8efb-3fe2c2abd71d/volumes" Oct 30 21:49:56.162972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.157377 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:56.162972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.157442 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:56.353976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.353095 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:56.706714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.706668 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:56.845972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.844383 6808 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:56.993024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:56.992911 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 21:49:57.041983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.034437 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc"] Oct 30 21:49:57.041983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.034683 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" containerID="cri-o://113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273" gracePeriod=30 Oct 30 21:49:57.081747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.079372 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:57.081747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.079432 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:57.095486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.095455 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h"] Oct 30 21:49:57.095865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.095840 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" containerID="cri-o://b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" gracePeriod=30 Oct 30 21:49:57.101513 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.101439 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d"] Oct 30 21:49:57.101878 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.101842 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" containerID="cri-o://c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3" gracePeriod=30 Oct 30 21:49:57.214974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:57.204215 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:58.050290 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.050253 6808 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 21:49:58.050495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.050312 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 21:49:58.816760 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.816714 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6"] Oct 30 21:49:58.817744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817710 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" Oct 30 21:49:58.817744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817740 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817776 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817782 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817796 6808 cpu_manager.go:414] "containerMap: removing container" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817803 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817824 6808 cpu_manager.go:414] "containerMap: removing container" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817829 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817839 6808 cpu_manager.go:414] "containerMap: removing container" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817843 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817853 6808 cpu_manager.go:414] "containerMap: removing container" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817857 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817868 6808 cpu_manager.go:414] "containerMap: removing container" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817873 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817899 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" Oct 30 21:49:58.817920 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.817903 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818032 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e78663-fd4e-451e-a3f4-380804008a8a" containerName="istio-proxy" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818044 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ac04d-5ae1-43f1-be29-5a08872cb890" containerName="istio-proxy" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818053 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c551711-d8a4-4824-aee5-d958286e738d" containerName="main" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818059 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818067 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="333f1d92-0c31-4a66-8efb-3fe2c2abd71d" containerName="istio-proxy" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818073 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="493dcec4-f99e-4ba5-9db1-87868c28af56" containerName="main" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818078 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d92887-ea5d-4ced-880e-de31ce124c20" containerName="istio-proxy" Oct 30 21:49:58.818524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.818085 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec09683-4e3f-4351-9faf-9cccccd36f9f" containerName="main" Oct 30 21:49:58.823298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.823277 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.836178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.836150 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6"] Oct 30 21:49:58.864266 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.864230 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc"] Oct 30 21:49:58.868807 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.868785 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.900375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.900310 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc"] Oct 30 21:49:58.957544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.957475 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.957896 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.957875 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.964036 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964000 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.964293 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964265 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.964370 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964338 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.964428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964381 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.964480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964460 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.964539 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964511 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.964592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964581 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qc5l\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.964880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.964855 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.965055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.965035 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxl9\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.965204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.965190 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.965363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.965349 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.965520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.965505 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:58.965660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.965642 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.965844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.965825 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.966099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.966078 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:58.966265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:58.966243 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.055350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.055315 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:49:59.055729 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.055709 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:49:59.056494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.056469 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:49:59.068416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068333 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.068416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068399 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068433 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068484 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068537 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068561 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068579 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068604 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.068626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068621 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068660 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068683 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068719 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qc5l\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068785 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068798 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068806 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxl9\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068893 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.068985 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069033 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.069033 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.069060 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.069496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.069217 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.069496 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.069444 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.069741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.069719 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.069899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.069871 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.070298 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.070277 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.070416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.070325 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.071146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.071115 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.071522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.071500 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.071670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.071647 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.072144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.072031 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.072312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.072292 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.073248 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.073225 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.073440 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.073423 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.083445 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.083424 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.086362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.085771 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.086362 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.086261 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qc5l\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.087779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.087756 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxl9\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.139787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.139750 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:49:59.206121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.206092 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:49:59.481175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.481151 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6"] Oct 30 21:49:59.482122 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:49:59.482079 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348cec2d_fbc4_4adb_b8f6_2e50d4e1aeab.slice/crio-a4935ee186bdbcf3fbcc3d58b501b92ffa9f906a8e5cc0db9ae7e6543ec28b1d WatchSource:0}: Error finding container a4935ee186bdbcf3fbcc3d58b501b92ffa9f906a8e5cc0db9ae7e6543ec28b1d: Status 404 returned error can't find the container with id a4935ee186bdbcf3fbcc3d58b501b92ffa9f906a8e5cc0db9ae7e6543ec28b1d Oct 30 21:49:59.485025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.484989 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:59.485100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.485076 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:59.485140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.485120 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:59.633159 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:49:59.633128 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4724778b_c4f9_4b2e_9f4d_e08933ea4a16.slice/crio-ba1d90cf71becd3bd86da842d285225b5120863e8b7145218abcbabdeee0a69c WatchSource:0}: Error finding container ba1d90cf71becd3bd86da842d285225b5120863e8b7145218abcbabdeee0a69c: Status 404 returned error can't find the container with id ba1d90cf71becd3bd86da842d285225b5120863e8b7145218abcbabdeee0a69c Oct 30 21:49:59.635471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.635447 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc"] Oct 30 21:49:59.636634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.636598 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:59.636721 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.636671 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:49:59.636779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:49:59.636715 6808 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:50:00.000845 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.000808 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf"] Oct 30 21:50:00.001411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.001389 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" event={"ID":"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab","Type":"ContainerStarted","Data":"c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c"} Oct 30 21:50:00.002128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.002039 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" event={"ID":"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab","Type":"ContainerStarted","Data":"a4935ee186bdbcf3fbcc3d58b501b92ffa9f906a8e5cc0db9ae7e6543ec28b1d"} Oct 30 21:50:00.002379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.001658 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" podUID="2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" containerName="maas-api" containerID="cri-o://49e797500efca3688a5be75f8bc1974528a2b0bbdf6b6f90bfbdef1676193777" gracePeriod=30 Oct 30 21:50:00.031984 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.028183 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podStartSLOduration=2.028164154 podStartE2EDuration="2.028164154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:00.0183488 +0000 UTC m=+1262.214039919" watchObservedRunningTime="2026-10-30 21:50:00.028164154 +0000 UTC m=+1262.223855274" Oct 30 21:50:00.085345 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.085311 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" event={"ID":"4724778b-c4f9-4b2e-9f4d-e08933ea4a16","Type":"ContainerStarted","Data":"2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea"} Oct 30 21:50:00.085621 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.085594 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" event={"ID":"4724778b-c4f9-4b2e-9f4d-e08933ea4a16","Type":"ContainerStarted","Data":"ba1d90cf71becd3bd86da842d285225b5120863e8b7145218abcbabdeee0a69c"} Oct 30 21:50:00.143328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.143299 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:50:00.143606 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.143584 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:00.143787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.143763 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:00.190090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.182590 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podStartSLOduration=2.182569134 podStartE2EDuration="2.182569134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:00.128655621 +0000 UTC m=+1262.324346740" watchObservedRunningTime="2026-10-30 21:50:00.182569134 +0000 UTC m=+1262.378260254" Oct 30 21:50:00.245028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.208013 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:50:00.245028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.208609 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:00.245028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:00.242139 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:01.017542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.017455 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:01.017542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.017526 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:01.120910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.120873 6808 generic.go:356] "Generic (PLEG): container finished" podID="2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" containerID="49e797500efca3688a5be75f8bc1974528a2b0bbdf6b6f90bfbdef1676193777" exitCode=0 Oct 30 21:50:01.121113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.121048 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" event={"ID":"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6","Type":"ContainerDied","Data":"49e797500efca3688a5be75f8bc1974528a2b0bbdf6b6f90bfbdef1676193777"} Oct 30 21:50:01.123214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.123180 6808 generic.go:356] "Generic (PLEG): container finished" podID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerID="cd7b5c2b769c4bb2c7d4edc893857d0d7a5bd81e96722154e09174253b18e7e3" exitCode=0 Oct 30 21:50:01.123973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.123922 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" event={"ID":"4b88d721-c32d-487a-b10d-1a5c48e940ce","Type":"ContainerDied","Data":"cd7b5c2b769c4bb2c7d4edc893857d0d7a5bd81e96722154e09174253b18e7e3"} Oct 30 21:50:01.155016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.154394 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:01.155016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.154462 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:01.215013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.214396 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:01.215013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.214452 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:01.312158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.312124 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:50:01.314694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.314673 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:50:01.422495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.422453 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.422756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.422721 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.434501 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.434472 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls\") pod \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " Oct 30 21:50:01.435075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.435037 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.435305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.435274 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs\") pod \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " Oct 30 21:50:01.435456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.435423 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m\" (UniqueName: \"kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m\") pod \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " Oct 30 21:50:01.435646 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.435632 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca\") pod \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\" (UID: \"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6\") " Oct 30 21:50:01.435820 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.435804 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.435995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.435944 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.436113 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.436100 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.436239 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.436227 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.436363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.436349 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.436595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.436571 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token\") pod \"4b88d721-c32d-487a-b10d-1a5c48e940ce\" (UID: \"4b88d721-c32d-487a-b10d-1a5c48e940ce\") " Oct 30 21:50:01.459723 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.427608 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:01.459723 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.427990 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:01.459723 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.444140 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca" pod "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" (UID: "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:01.460250 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.460056 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:01.460659 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.460383 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:01.463549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.463525 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:01.520812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.465063 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:01.522300 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.472176 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "kube-api-access-hpgbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:01.522732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.490171 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:01.523288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.523082 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:01.523288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.500077 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:01.523288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.500145 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs" pod "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" (UID: "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:01.591976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.550782 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.592417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.592305 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.592565 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.592552 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.611057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.611023 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.611293 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.611273 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b88d721-c32d-487a-b10d-1a5c48e940ce-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.611392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.611379 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.611498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.611485 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.625055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.625016 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.625252 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.625235 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgbw\" (UniqueName: \"kubernetes.io/projected/4b88d721-c32d-487a-b10d-1a5c48e940ce-kube-api-access-hpgbw\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.634978 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.625353 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.635180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.566389 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy" pod "4b88d721-c32d-487a-b10d-1a5c48e940ce" (UID: "4b88d721-c32d-487a-b10d-1a5c48e940ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:01.635327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.567148 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m" pod "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" (UID: "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6"). InnerVolumeSpecName "kube-api-access-h2j6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:01.740574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.740453 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2j6m\" (UniqueName: \"kubernetes.io/projected/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-kube-api-access-h2j6m\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.740574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.740495 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b88d721-c32d-487a-b10d-1a5c48e940ce-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.770013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.769181 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls" pod "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" (UID: "2bf2d7da-0533-4ea8-ae0e-1de61082c5a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:01.847802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:01.847761 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:02.134886 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.134858 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" Oct 30 21:50:02.135304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.134857 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf" event={"ID":"2bf2d7da-0533-4ea8-ae0e-1de61082c5a6","Type":"ContainerDied","Data":"cb336d161c90141ad667648abdc9d7b9b149dbe46bd7d7963e7a82cdf5219c79"} Oct 30 21:50:02.135304 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.135041 6808 scope.go:122] "RemoveContainer" containerID="49e797500efca3688a5be75f8bc1974528a2b0bbdf6b6f90bfbdef1676193777" Oct 30 21:50:02.137023 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.136998 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" event={"ID":"4b88d721-c32d-487a-b10d-1a5c48e940ce","Type":"ContainerDied","Data":"f70586257499b64b04e6bf3988917ffd6c35c5280b94066144a5d5f62ccfdba4"} Oct 30 21:50:02.137355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.137021 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x" Oct 30 21:50:02.152986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.152941 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:02.153175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.153151 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:02.207987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.207277 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:02.207987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.207318 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:02.321893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.321790 6808 scope.go:122] "RemoveContainer" containerID="cd7b5c2b769c4bb2c7d4edc893857d0d7a5bd81e96722154e09174253b18e7e3" Oct 30 21:50:02.338566 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.338545 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x"] Oct 30 21:50:02.625529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.440074 6808 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:50:02.625529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.440149 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:50:02.628494 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.626860 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-m562x"] Oct 30 21:50:02.635569 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.635540 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf"] Oct 30 21:50:02.635704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:02.635591 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ba0bd-5688bcf869-4jmxf"] Oct 30 21:50:03.135992 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.134604 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:50:03.137007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.136986 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:50:03.208275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.208233 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:03.208471 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.208308 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:03.210414 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.210342 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:03.210934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.210903 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:03.215918 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.215473 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:50:03.223904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.223716 6808 generic.go:356] "Generic (PLEG): container finished" podID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerID="113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273" exitCode=0 Oct 30 21:50:03.223904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.223859 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" event={"ID":"d367b0db-fb5d-4255-ac52-7a6c07d48275","Type":"ContainerDied","Data":"113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273"} Oct 30 21:50:03.223904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.223874 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" Oct 30 21:50:03.224174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.223914 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc" event={"ID":"d367b0db-fb5d-4255-ac52-7a6c07d48275","Type":"ContainerDied","Data":"acb7618ea2094ede0c4bc0c3e1989396b6ee9fc6ce18fc32fae7ad57de6030da"} Oct 30 21:50:03.224174 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.223977 6808 scope.go:122] "RemoveContainer" containerID="113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273" Oct 30 21:50:03.227417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.226994 6808 generic.go:356] "Generic (PLEG): container finished" podID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerID="c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3" exitCode=0 Oct 30 21:50:03.227417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.227057 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" event={"ID":"3cc2476c-7f34-4ed3-b880-b797ee00aa02","Type":"ContainerDied","Data":"c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3"} Oct 30 21:50:03.227417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.227088 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" event={"ID":"3cc2476c-7f34-4ed3-b880-b797ee00aa02","Type":"ContainerDied","Data":"47238b326a963c88cb65b90ce8ba76209393e1b0c945a992fef97e6d514545e8"} Oct 30 21:50:03.227417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.227138 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d" Oct 30 21:50:03.245969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.238827 6808 generic.go:356] "Generic (PLEG): container finished" podID="60c75041-02c6-4326-ae01-bb651acf4497" containerID="b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" exitCode=0 Oct 30 21:50:03.245969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.238898 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" event={"ID":"60c75041-02c6-4326-ae01-bb651acf4497","Type":"ContainerDied","Data":"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603"} Oct 30 21:50:03.245969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.238936 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" event={"ID":"60c75041-02c6-4326-ae01-bb651acf4497","Type":"ContainerDied","Data":"90e0d429fa91de3367397817768e0bb53ff79de4a6ec507adf1b8911442dcb34"} Oct 30 21:50:03.245969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.239022 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h" Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316452 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316516 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316600 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316646 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316684 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316726 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316826 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316871 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.316908 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317066 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317114 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317174 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317215 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317361 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317399 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy\") pod \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\" (UID: \"3cc2476c-7f34-4ed3-b880-b797ee00aa02\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317453 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317496 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.318757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.317532 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo\") pod \"d367b0db-fb5d-4255-ac52-7a6c07d48275\" (UID: \"d367b0db-fb5d-4255-ac52-7a6c07d48275\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.333662 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.334057 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.336641 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.344576 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.347599 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.351618 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.352259 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.354977 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.355320 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.355420 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.362165 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.375158 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.417124 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.425840 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.425930 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.425993 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.426030 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.426212 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.426288 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.426403 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.426492 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.426547 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo\") pod \"60c75041-02c6-4326-ae01-bb651acf4497\" (UID: \"60c75041-02c6-4326-ae01-bb651acf4497\") " Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427542 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427575 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427595 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427613 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427627 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427645 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427664 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427676 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427689 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427705 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427723 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427735 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427749 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367b0db-fb5d-4255-ac52-7a6c07d48275-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.427984 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.501237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.487583 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.555141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.554711 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:03.555141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.554813 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.586045 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.590015 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.591481 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601310 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601336 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601354 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601369 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601383 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601396 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75041-02c6-4326-ae01-bb651acf4497-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.601411 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc2476c-7f34-4ed3-b880-b797ee00aa02-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.606295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.602133 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:03.608875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.607102 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "kube-api-access-fr2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.608875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.607557 6808 scope.go:122] "RemoveContainer" containerID="113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273" Oct 30 21:50:03.609903 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:03.609430 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273\": container with ID starting with 113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273 not found: ID does not exist" containerID="113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273" Oct 30 21:50:03.609903 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.609466 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273"} err="failed to get container status \"113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273\": rpc error: code = NotFound desc = could not find container \"113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273\": container with ID starting with 113c69d2a078304a5825b5f337afc551b68ceea12f0b0a0ffd94a56c6194f273 not found: ID does not exist" Oct 30 21:50:03.609903 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.609484 6808 scope.go:122] "RemoveContainer" containerID="c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3" Oct 30 21:50:03.613736 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.613708 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:03.614617 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.614574 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "kube-api-access-8njwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.619004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.615554 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv" pod "3cc2476c-7f34-4ed3-b880-b797ee00aa02" (UID: "3cc2476c-7f34-4ed3-b880-b797ee00aa02"). InnerVolumeSpecName "kube-api-access-45tdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.619004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.617540 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token" pod "d367b0db-fb5d-4255-ac52-7a6c07d48275" (UID: "d367b0db-fb5d-4255-ac52-7a6c07d48275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.619004 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.617568 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token" pod "60c75041-02c6-4326-ae01-bb651acf4497" (UID: "60c75041-02c6-4326-ae01-bb651acf4497"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.623309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.623289 6808 scope.go:122] "RemoveContainer" containerID="c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3" Oct 30 21:50:03.623664 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:03.623640 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3\": container with ID starting with c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3 not found: ID does not exist" containerID="c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3" Oct 30 21:50:03.623782 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.623667 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3"} err="failed to get container status \"c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3\": rpc error: code = NotFound desc = could not find container \"c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3\": container with ID starting with c3fa949e74203bd57b053a09e1fa947e583a63e30913149205c269e266037ab3 not found: ID does not exist" Oct 30 21:50:03.623782 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.623685 6808 scope.go:122] "RemoveContainer" containerID="b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" Oct 30 21:50:03.754578 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.754464 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.795137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.795087 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.795730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.795626 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8njwg\" (UniqueName: \"kubernetes.io/projected/60c75041-02c6-4326-ae01-bb651acf4497-kube-api-access-8njwg\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.795730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.795680 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75041-02c6-4326-ae01-bb651acf4497-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.795730 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.795699 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2qw\" (UniqueName: \"kubernetes.io/projected/d367b0db-fb5d-4255-ac52-7a6c07d48275-kube-api-access-fr2qw\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.796125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.796057 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45tdv\" (UniqueName: \"kubernetes.io/projected/3cc2476c-7f34-4ed3-b880-b797ee00aa02-kube-api-access-45tdv\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.796125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.796085 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75041-02c6-4326-ae01-bb651acf4497-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.797264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.797204 6808 scope.go:122] "RemoveContainer" containerID="b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" Oct 30 21:50:03.885972 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:03.844271 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603\": container with ID starting with b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603 not found: ID does not exist" containerID="b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" Oct 30 21:50:03.885972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.844325 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603"} err="failed to get container status \"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603\": rpc error: code = NotFound desc = could not find container \"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603\": container with ID starting with b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603 not found: ID does not exist" Oct 30 21:50:03.890013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:03.889019 6808 scope.go:122] "RemoveContainer" containerID="b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" Oct 30 21:50:03.963367 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:03.963240 6808 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603\": rpc error: code = NotFound desc = could not find container \"b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603\": container with ID starting with b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603 not found: ID does not exist" containerID="b02999fc8b169770e6bef406b67123077e31712c5b3e75b0d6a02ff8c7929603" Oct 30 21:50:04.093971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.092073 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" path="/var/lib/kubelet/pods/2bf2d7da-0533-4ea8-ae0e-1de61082c5a6/volumes" Oct 30 21:50:04.093971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.092835 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" path="/var/lib/kubelet/pods/4b88d721-c32d-487a-b10d-1a5c48e940ce/volumes" Oct 30 21:50:04.116388 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.111768 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc"] Oct 30 21:50:04.116388 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.111872 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-cc7wc"] Oct 30 21:50:04.160350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.151408 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h"] Oct 30 21:50:04.160350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.151746 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:04.160350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.151796 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:04.216569 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.216065 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:04.216569 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.216138 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:04.271563 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.270072 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-8g67h"] Oct 30 21:50:04.318693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.317579 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d"] Oct 30 21:50:04.370669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:04.370636 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-84j7d"] Oct 30 21:50:05.150005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:05.149693 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:05.150005 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:05.149771 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:05.220411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:05.220176 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:05.220411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:05.220234 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:05.721498 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:05.721466 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:50:06.038815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.038739 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" path="/var/lib/kubelet/pods/3cc2476c-7f34-4ed3-b880-b797ee00aa02/volumes" Oct 30 21:50:06.039296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.039282 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c75041-02c6-4326-ae01-bb651acf4497" path="/var/lib/kubelet/pods/60c75041-02c6-4326-ae01-bb651acf4497/volumes" Oct 30 21:50:06.039767 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.039753 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" path="/var/lib/kubelet/pods/d367b0db-fb5d-4255-ac52-7a6c07d48275/volumes" Oct 30 21:50:06.164973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.164341 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:50:06.164973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.164408 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 21:50:06.207676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.207637 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:06.207884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:06.207698 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:07.145097 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:07.145057 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:50:07.145516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:07.145383 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:50:07.146622 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:07.146602 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:50:07.207923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:07.207892 6808 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:50:07.208162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:07.208134 6808 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 21:50:08.234976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:08.233250 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:50:08.234976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:08.234772 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:50:08.235996 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:08.235973 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:50:12.036463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:12.036377 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6"] Oct 30 21:50:12.036922 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:12.036783 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" containerID="cri-o://c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c" gracePeriod=30 Oct 30 21:50:12.072899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:12.072869 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb"] Oct 30 21:50:12.073102 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:12.073082 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" containerID="cri-o://3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6" gracePeriod=30 Oct 30 21:50:12.077981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:12.077931 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc"] Oct 30 21:50:12.078171 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:12.078144 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" containerID="cri-o://2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea" gracePeriod=30 Oct 30 21:50:17.497364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.497329 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:50:17.564685 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.564654 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:50:17.566696 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.566668 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:50:17.624689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624658 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.624885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624697 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.624885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624739 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.624885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624776 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.624885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624813 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.624885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624844 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.624885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624880 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624919 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.624963 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625003 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625038 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625101 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625132 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625162 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625208 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625184 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625271 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625307 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625374 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625414 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625470 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625544 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625586 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625585 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625373 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625598 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625652 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625665 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625735 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625786 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625921 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625921 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9\") pod \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\" (UID: \"4724778b-c4f9-4b2e-9f4d-e08933ea4a16\") " Oct 30 21:50:17.625981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625977 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.626433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.625996 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert\") pod \"4177f737-6bc7-4647-b1d8-351e85c33cee\" (UID: \"4177f737-6bc7-4647-b1d8-351e85c33cee\") " Oct 30 21:50:17.626433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.626025 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.626433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.626114 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs\") pod \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\" (UID: \"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab\") " Oct 30 21:50:17.626433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.626141 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.626433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.626183 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.626659 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.626633 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.626881 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.626857 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.627052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627033 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627058 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627073 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627087 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627100 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627115 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627129 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627142 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627156 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627169 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627184 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.627818 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627790 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.627922 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.627806 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.632137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.629605 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.632137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.629769 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.636363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636318 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.636363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636332 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:17.636363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636322 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "kube-api-access-4qxl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.636574 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636380 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.636808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636706 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:17.636808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636729 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:17.636808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636751 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.636808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636765 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.636808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.636768 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "kube-api-access-2qc5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.637188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.637170 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token" pod "4724778b-c4f9-4b2e-9f4d-e08933ea4a16" (UID: "4724778b-c4f9-4b2e-9f4d-e08933ea4a16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.637556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.637534 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy" pod "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" (UID: "348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.638842 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.638821 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5" pod "4177f737-6bc7-4647-b1d8-351e85c33cee" (UID: "4177f737-6bc7-4647-b1d8-351e85c33cee"). InnerVolumeSpecName "kube-api-access-67ln5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.651769 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.651730 6808 generic.go:356] "Generic (PLEG): container finished" podID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerID="c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c" exitCode=0 Oct 30 21:50:17.651887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.651780 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" Oct 30 21:50:17.652238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.651991 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" event={"ID":"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab","Type":"ContainerDied","Data":"c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c"} Oct 30 21:50:17.652238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.652052 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6" event={"ID":"348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab","Type":"ContainerDied","Data":"a4935ee186bdbcf3fbcc3d58b501b92ffa9f906a8e5cc0db9ae7e6543ec28b1d"} Oct 30 21:50:17.652238 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.652101 6808 scope.go:122] "RemoveContainer" containerID="c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c" Oct 30 21:50:17.655144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.654506 6808 generic.go:356] "Generic (PLEG): container finished" podID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerID="3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6" exitCode=0 Oct 30 21:50:17.655144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.654590 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" event={"ID":"4177f737-6bc7-4647-b1d8-351e85c33cee","Type":"ContainerDied","Data":"3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6"} Oct 30 21:50:17.655144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.654632 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" event={"ID":"4177f737-6bc7-4647-b1d8-351e85c33cee","Type":"ContainerDied","Data":"82042d5885833de51ad27f1a8a68058e240837e8a93f9228143da1b6c92491ce"} Oct 30 21:50:17.655144 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.654716 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb" Oct 30 21:50:17.661507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.661483 6808 generic.go:356] "Generic (PLEG): container finished" podID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerID="2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea" exitCode=0 Oct 30 21:50:17.661599 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.661551 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" event={"ID":"4724778b-c4f9-4b2e-9f4d-e08933ea4a16","Type":"ContainerDied","Data":"2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea"} Oct 30 21:50:17.661599 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.661588 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" event={"ID":"4724778b-c4f9-4b2e-9f4d-e08933ea4a16","Type":"ContainerDied","Data":"ba1d90cf71becd3bd86da842d285225b5120863e8b7145218abcbabdeee0a69c"} Oct 30 21:50:17.661725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.661654 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc" Oct 30 21:50:17.679417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.679340 6808 scope.go:122] "RemoveContainer" containerID="c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c" Oct 30 21:50:17.681261 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:17.681235 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c\": container with ID starting with c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c not found: ID does not exist" containerID="c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c" Oct 30 21:50:17.681364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.681272 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c"} err="failed to get container status \"c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c\": rpc error: code = NotFound desc = could not find container \"c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c\": container with ID starting with c7dd610180258f96c040637ece1573750dc4e19d20d70279d8b74023f9401e5c not found: ID does not exist" Oct 30 21:50:17.681364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.681295 6808 scope.go:122] "RemoveContainer" containerID="3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6" Oct 30 21:50:17.692422 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.692365 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6"] Oct 30 21:50:17.695853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.695833 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qqgr6"] Oct 30 21:50:17.701072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.701055 6808 scope.go:122] "RemoveContainer" containerID="25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7" Oct 30 21:50:17.705328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.705306 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb"] Oct 30 21:50:17.709538 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.709518 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-pgzkb"] Oct 30 21:50:17.710868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.710851 6808 scope.go:122] "RemoveContainer" containerID="3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6" Oct 30 21:50:17.711179 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:17.711162 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6\": container with ID starting with 3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6 not found: ID does not exist" containerID="3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6" Oct 30 21:50:17.711246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.711186 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6"} err="failed to get container status \"3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6\": rpc error: code = NotFound desc = could not find container \"3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6\": container with ID starting with 3053bdd096d836984b25dbd4740e1491cc75611d06331bb330f9a43cf91b99a6 not found: ID does not exist" Oct 30 21:50:17.711246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.711201 6808 scope.go:122] "RemoveContainer" containerID="25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7" Oct 30 21:50:17.711420 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:17.711406 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7\": container with ID starting with 25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7 not found: ID does not exist" containerID="25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7" Oct 30 21:50:17.711457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.711424 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7"} err="failed to get container status \"25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7\": rpc error: code = NotFound desc = could not find container \"25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7\": container with ID starting with 25284557f2bb0cefe36d6a00c7bf9b904398ce08d515b1e00956a1c14e66e2e7 not found: ID does not exist" Oct 30 21:50:17.711457 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.711435 6808 scope.go:122] "RemoveContainer" containerID="2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea" Oct 30 21:50:17.720898 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.720879 6808 scope.go:122] "RemoveContainer" containerID="2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea" Oct 30 21:50:17.721193 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:17.721170 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea\": container with ID starting with 2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea not found: ID does not exist" containerID="2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea" Oct 30 21:50:17.721246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.721204 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea"} err="failed to get container status \"2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea\": rpc error: code = NotFound desc = could not find container \"2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea\": container with ID starting with 2765fa45be5a8eedd8265a9742b20ee5158fe7988c6df1792030e7d67d070eea not found: ID does not exist" Oct 30 21:50:17.722923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.722904 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc"] Oct 30 21:50:17.726776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.726755 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-qkmzc"] Oct 30 21:50:17.728343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728325 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728346 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728356 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qc5l\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-kube-api-access-2qc5l\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728366 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728375 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728383 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728391 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728399 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4177f737-6bc7-4647-b1d8-351e85c33cee-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728409 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728417 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728425 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxl9\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-kube-api-access-4qxl9\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728433 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4177f737-6bc7-4647-b1d8-351e85c33cee-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728441 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724778b-c4f9-4b2e-9f4d-e08933ea4a16-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728450 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ln5\" (UniqueName: \"kubernetes.io/projected/4177f737-6bc7-4647-b1d8-351e85c33cee-kube-api-access-67ln5\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728460 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.728573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.728468 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.930976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.930858 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" path="/var/lib/kubelet/pods/348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab/volumes" Oct 30 21:50:17.931418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.931399 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" path="/var/lib/kubelet/pods/4177f737-6bc7-4647-b1d8-351e85c33cee/volumes" Oct 30 21:50:17.931887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:17.931874 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" path="/var/lib/kubelet/pods/4724778b-c4f9-4b2e-9f4d-e08933ea4a16/volumes" Oct 30 21:50:19.439624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:19.439537 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d"] Oct 30 21:50:21.508704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:21.508671 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t"] Oct 30 21:50:21.509379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:21.509352 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" podUID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" containerName="maas-api" containerID="cri-o://897c2d636e09e6743ce622bde8fc07a687711f102fd4f4640f0fc9d543e8a0ca" gracePeriod=30 Oct 30 21:50:21.663374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:21.663331 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" podUID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": dial tcp 10.134.0.124:8443: connect: connection refused" Oct 30 21:50:22.070537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.070366 6808 generic.go:356] "Generic (PLEG): container finished" podID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" containerID="897c2d636e09e6743ce622bde8fc07a687711f102fd4f4640f0fc9d543e8a0ca" exitCode=0 Oct 30 21:50:22.070537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.070467 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" event={"ID":"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57","Type":"ContainerDied","Data":"897c2d636e09e6743ce622bde8fc07a687711f102fd4f4640f0fc9d543e8a0ca"} Oct 30 21:50:22.183043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.183015 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:50:22.205067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.205028 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb\" (UniqueName: \"kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb\") pod \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " Oct 30 21:50:22.218126 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.218093 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb" pod "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" (UID: "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57"). InnerVolumeSpecName "kube-api-access-cxsmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:22.306882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.306845 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca\") pod \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " Oct 30 21:50:22.307080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.306997 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs\") pod \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " Oct 30 21:50:22.307185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.307154 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls\") pod \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\" (UID: \"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57\") " Oct 30 21:50:22.307253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.307196 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca" pod "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" (UID: "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:22.307935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.307905 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:22.307935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.307936 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsmb\" (UniqueName: \"kubernetes.io/projected/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-kube-api-access-cxsmb\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:22.309428 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.309407 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls" pod "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" (UID: "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:22.309604 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.309494 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs" pod "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" (UID: "8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:22.427979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.413078 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:22.427979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:22.413128 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:23.117970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:23.110136 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" event={"ID":"8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57","Type":"ContainerDied","Data":"0c62581da5df6f94196a5fd408ead7f090d79694f49641232e77f38811c2ffaf"} Oct 30 21:50:23.117970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:23.110215 6808 scope.go:122] "RemoveContainer" containerID="897c2d636e09e6743ce622bde8fc07a687711f102fd4f4640f0fc9d543e8a0ca" Oct 30 21:50:23.117970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:23.110411 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t" Oct 30 21:50:23.314522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:23.314441 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t"] Oct 30 21:50:23.338373 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:23.317760 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-950784-748cc4875c-q7g6t"] Oct 30 21:50:23.929994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:23.929935 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" path="/var/lib/kubelet/pods/8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57/volumes" Oct 30 21:50:24.445977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.445909 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" podUID="9b7dfc83-82fe-43b7-af39-bcad0c9a7957" containerName="payload-pre-processing" containerID="cri-o://310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65" gracePeriod=25 Oct 30 21:50:24.489594 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:24.489567 6808 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7dfc83_82fe_43b7_af39_bcad0c9a7957.slice/crio-310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:50:24.720148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.720120 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:50:24.774595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.774571 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume\") pod \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " Oct 30 21:50:24.774727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.774686 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr\" (UniqueName: \"kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr\") pod \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\" (UID: \"9b7dfc83-82fe-43b7-af39-bcad0c9a7957\") " Oct 30 21:50:24.774911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.774890 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume" pod "9b7dfc83-82fe-43b7-af39-bcad0c9a7957" (UID: "9b7dfc83-82fe-43b7-af39-bcad0c9a7957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:24.775409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.775388 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:24.776726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.776707 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr" pod "9b7dfc83-82fe-43b7-af39-bcad0c9a7957" (UID: "9b7dfc83-82fe-43b7-af39-bcad0c9a7957"). InnerVolumeSpecName "kube-api-access-vbckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:24.876532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:24.876496 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbckr\" (UniqueName: \"kubernetes.io/projected/9b7dfc83-82fe-43b7-af39-bcad0c9a7957-kube-api-access-vbckr\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:25.126999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.126909 6808 generic.go:356] "Generic (PLEG): container finished" podID="9b7dfc83-82fe-43b7-af39-bcad0c9a7957" containerID="310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65" exitCode=0 Oct 30 21:50:25.126999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.126974 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" event={"ID":"9b7dfc83-82fe-43b7-af39-bcad0c9a7957","Type":"ContainerDied","Data":"310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65"} Oct 30 21:50:25.126999 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.126990 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" Oct 30 21:50:25.127236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.127029 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d" event={"ID":"9b7dfc83-82fe-43b7-af39-bcad0c9a7957","Type":"ContainerDied","Data":"fbaa6387060826259a4d2885be9e7f3964371aefb54193abf7d39f57366bac2e"} Oct 30 21:50:25.127236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.127050 6808 scope.go:122] "RemoveContainer" containerID="310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65" Oct 30 21:50:25.141684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.140450 6808 scope.go:122] "RemoveContainer" containerID="310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65" Oct 30 21:50:25.141771 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:25.141699 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65\": container with ID starting with 310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65 not found: ID does not exist" containerID="310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65" Oct 30 21:50:25.141771 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.141725 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65"} err="failed to get container status \"310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65\": rpc error: code = NotFound desc = could not find container \"310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65\": container with ID starting with 310a33b6364795374a7afd08d0edb49762b23031cd5ab92913be89806c0acc65 not found: ID does not exist" Oct 30 21:50:25.156607 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.156578 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d"] Oct 30 21:50:25.160803 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.160784 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1f725d-68d665b4cf-bcf8d"] Oct 30 21:50:25.929032 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:25.929002 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7dfc83-82fe-43b7-af39-bcad0c9a7957" path="/var/lib/kubelet/pods/9b7dfc83-82fe-43b7-af39-bcad0c9a7957/volumes" Oct 30 21:50:34.153245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:34.122508 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn"] Oct 30 21:50:34.153245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:34.144746 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz"] Oct 30 21:50:35.044969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:35.040067 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9"] Oct 30 21:50:35.044969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:35.040330 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" containerID="cri-o://1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a" gracePeriod=30 Oct 30 21:50:35.704122 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:35.704082 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:50:38.805940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:38.805895 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8"] Oct 30 21:50:38.811651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:38.811588 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" podUID="bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" containerName="maas-api" containerID="cri-o://1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3" gracePeriod=30 Oct 30 21:50:39.156974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.148085 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" podUID="a2e60288-7a9a-449f-85c7-897f9b5a1b99" containerName="payload-pre-processing" containerID="cri-o://a2e1f1ff199e1c5513abf96cb33ed32005b80ce7ebe4d895c0a7673ab17d670f" gracePeriod=25 Oct 30 21:50:39.188860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.146128 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" podUID="815fbaae-7bbc-4560-928b-269a8b34634f" containerName="payload-processing" containerID="cri-o://671581d11695ce5aa0ca65a75fff1b8822b8337fb20920afc498541b26b0c546" gracePeriod=25 Oct 30 21:50:39.221378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.221012 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:39.221378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.221096 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:39.230972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.230135 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:50:39.296971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.288966 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\") pod \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " Oct 30 21:50:39.296971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.289131 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d\" (UniqueName: \"kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d\") pod \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " Oct 30 21:50:39.296971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.289233 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca\") pod \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " Oct 30 21:50:39.296971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.289346 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs\") pod \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\" (UID: \"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc\") " Oct 30 21:50:39.311261 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.310074 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs" pod "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" (UID: "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:39.315125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.314071 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca" pod "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" (UID: "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:39.317538 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.316058 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d" pod "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" (UID: "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc"). InnerVolumeSpecName "kube-api-access-xc59d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:39.327473 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.327138 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls" pod "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" (UID: "bfd168fc-9a7b-4dbc-801a-a30cf715fdbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:39.396438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.396244 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.396438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.396281 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.396438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.396296 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc59d\" (UniqueName: \"kubernetes.io/projected/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-kube-api-access-xc59d\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.396438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.396312 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.440987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.438062 6808 generic.go:356] "Generic (PLEG): container finished" podID="bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" containerID="1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3" exitCode=0 Oct 30 21:50:39.440987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.438142 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" event={"ID":"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc","Type":"ContainerDied","Data":"1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3"} Oct 30 21:50:39.440987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.438181 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" event={"ID":"bfd168fc-9a7b-4dbc-801a-a30cf715fdbc","Type":"ContainerDied","Data":"fb7c990027d6067aa39d528f5e208a10ee31a7a00b7f6852038580f8a08ab468"} Oct 30 21:50:39.440987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.438207 6808 scope.go:122] "RemoveContainer" containerID="1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3" Oct 30 21:50:39.440987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.438326 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8" Oct 30 21:50:39.444522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.444322 6808 generic.go:356] "Generic (PLEG): container finished" podID="815fbaae-7bbc-4560-928b-269a8b34634f" containerID="671581d11695ce5aa0ca65a75fff1b8822b8337fb20920afc498541b26b0c546" exitCode=0 Oct 30 21:50:39.444522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.444438 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" event={"ID":"815fbaae-7bbc-4560-928b-269a8b34634f","Type":"ContainerDied","Data":"671581d11695ce5aa0ca65a75fff1b8822b8337fb20920afc498541b26b0c546"} Oct 30 21:50:39.448302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.447756 6808 generic.go:356] "Generic (PLEG): container finished" podID="a2e60288-7a9a-449f-85c7-897f9b5a1b99" containerID="a2e1f1ff199e1c5513abf96cb33ed32005b80ce7ebe4d895c0a7673ab17d670f" exitCode=0 Oct 30 21:50:39.448302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.447922 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" event={"ID":"a2e60288-7a9a-449f-85c7-897f9b5a1b99","Type":"ContainerDied","Data":"a2e1f1ff199e1c5513abf96cb33ed32005b80ce7ebe4d895c0a7673ab17d670f"} Oct 30 21:50:39.456483 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.456462 6808 scope.go:122] "RemoveContainer" containerID="1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3" Oct 30 21:50:39.456747 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:39.456726 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3\": container with ID starting with 1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3 not found: ID does not exist" containerID="1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3" Oct 30 21:50:39.456811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.456756 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3"} err="failed to get container status \"1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3\": rpc error: code = NotFound desc = could not find container \"1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3\": container with ID starting with 1820d59a244a205066b3223b7834191b2572268464f5b2bfd65ffb43351e47d3 not found: ID does not exist" Oct 30 21:50:39.470091 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.470046 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8"] Oct 30 21:50:39.477969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.477379 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8144d-7d74949dc8-vkzf8"] Oct 30 21:50:39.577576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.577548 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:50:39.601732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.601121 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s\" (UniqueName: \"kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s\") pod \"815fbaae-7bbc-4560-928b-269a8b34634f\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " Oct 30 21:50:39.601732 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.601367 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume\") pod \"815fbaae-7bbc-4560-928b-269a8b34634f\" (UID: \"815fbaae-7bbc-4560-928b-269a8b34634f\") " Oct 30 21:50:39.603034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.602733 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume" pod "815fbaae-7bbc-4560-928b-269a8b34634f" (UID: "815fbaae-7bbc-4560-928b-269a8b34634f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:39.606365 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.606343 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s" pod "815fbaae-7bbc-4560-928b-269a8b34634f" (UID: "815fbaae-7bbc-4560-928b-269a8b34634f"). InnerVolumeSpecName "kube-api-access-8jx6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:39.647071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.647018 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:50:39.702844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.702817 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume\") pod \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " Oct 30 21:50:39.703197 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.703162 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf\" (UniqueName: \"kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf\") pod \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\" (UID: \"a2e60288-7a9a-449f-85c7-897f9b5a1b99\") " Oct 30 21:50:39.703302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.703238 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume" pod "a2e60288-7a9a-449f-85c7-897f9b5a1b99" (UID: "a2e60288-7a9a-449f-85c7-897f9b5a1b99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:39.703751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.703730 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e60288-7a9a-449f-85c7-897f9b5a1b99-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.703751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.703752 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815fbaae-7bbc-4560-928b-269a8b34634f-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.703914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.703762 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jx6s\" (UniqueName: \"kubernetes.io/projected/815fbaae-7bbc-4560-928b-269a8b34634f-kube-api-access-8jx6s\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.705293 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.705266 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf" pod "a2e60288-7a9a-449f-85c7-897f9b5a1b99" (UID: "a2e60288-7a9a-449f-85c7-897f9b5a1b99"). InnerVolumeSpecName "kube-api-access-lfhhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:39.804980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.804913 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhhf\" (UniqueName: \"kubernetes.io/projected/a2e60288-7a9a-449f-85c7-897f9b5a1b99-kube-api-access-lfhhf\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:39.929571 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:39.929495 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" path="/var/lib/kubelet/pods/bfd168fc-9a7b-4dbc-801a-a30cf715fdbc/volumes" Oct 30 21:50:40.457391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.457360 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" Oct 30 21:50:40.457593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.457385 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn" event={"ID":"815fbaae-7bbc-4560-928b-269a8b34634f","Type":"ContainerDied","Data":"5acda97de50a6a8964ad1d73a97c7f5dc2a0b88cf02272f659bba25ba62a93dd"} Oct 30 21:50:40.457593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.457457 6808 scope.go:122] "RemoveContainer" containerID="671581d11695ce5aa0ca65a75fff1b8822b8337fb20920afc498541b26b0c546" Oct 30 21:50:40.458880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.458858 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" event={"ID":"a2e60288-7a9a-449f-85c7-897f9b5a1b99","Type":"ContainerDied","Data":"05372b7f59bc1fc426e00cd00999df9352d7b86d9e962bd5329dc0a9d7446c19"} Oct 30 21:50:40.458998 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.458983 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz" Oct 30 21:50:40.468051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.468031 6808 scope.go:122] "RemoveContainer" containerID="a2e1f1ff199e1c5513abf96cb33ed32005b80ce7ebe4d895c0a7673ab17d670f" Oct 30 21:50:40.478970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.478933 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn"] Oct 30 21:50:40.483044 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.483017 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2b36c1-6fd6c4d4d5-wnhkn"] Oct 30 21:50:40.498480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.498459 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz"] Oct 30 21:50:40.503521 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.503500 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2b36c1-7c9c855b7-mghvz"] Oct 30 21:50:40.715861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.715765 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg"] Oct 30 21:50:40.764883 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.764856 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:50:40.824725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824632 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.824725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824670 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.824917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824737 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.824917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824778 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.824917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824888 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.824917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824904 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.825101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824931 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.825101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.824988 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.825101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.825094 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo\") pod \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\" (UID: \"cd168855-dfa6-412d-a18b-03a29a6bb7ca\") " Oct 30 21:50:40.825542 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.825510 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:40.825734 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.825687 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:40.825830 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.825783 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:40.825997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.825961 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:40.826278 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.826254 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:40.826927 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.826903 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.827013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.826933 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.827013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.826971 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.827013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.826987 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.827013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.827000 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.828101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.828072 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:40.828218 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.828168 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:40.828282 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.828229 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:40.829149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.829112 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v" pod "cd168855-dfa6-412d-a18b-03a29a6bb7ca" (UID: "cd168855-dfa6-412d-a18b-03a29a6bb7ca"). InnerVolumeSpecName "kube-api-access-ffz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:40.928780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.928747 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffz5v\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-kube-api-access-ffz5v\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.928780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.928776 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.928780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.928786 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:40.929021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:40.928795 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd168855-dfa6-412d-a18b-03a29a6bb7ca-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:41.468011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.467974 6808 generic.go:356] "Generic (PLEG): container finished" podID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerID="1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a" exitCode=0 Oct 30 21:50:41.468011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.468017 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" Oct 30 21:50:41.468526 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.468062 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" event={"ID":"cd168855-dfa6-412d-a18b-03a29a6bb7ca","Type":"ContainerDied","Data":"1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a"} Oct 30 21:50:41.468526 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.468116 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9" event={"ID":"cd168855-dfa6-412d-a18b-03a29a6bb7ca","Type":"ContainerDied","Data":"25856bdeb1df7475e188c1225e8d932a19ff594442dadd7d5648eba3f0bee595"} Oct 30 21:50:41.468526 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.468144 6808 scope.go:122] "RemoveContainer" containerID="1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a" Oct 30 21:50:41.482265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.482240 6808 scope.go:122] "RemoveContainer" containerID="c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1" Oct 30 21:50:41.497017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.496990 6808 scope.go:122] "RemoveContainer" containerID="1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a" Oct 30 21:50:41.497363 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:41.497338 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a\": container with ID starting with 1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a not found: ID does not exist" containerID="1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a" Oct 30 21:50:41.497523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.497378 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a"} err="failed to get container status \"1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a\": rpc error: code = NotFound desc = could not find container \"1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a\": container with ID starting with 1380d51a2d00220380bc7c53aa4c010613174f94fa8893b7f6dd854b3301787a not found: ID does not exist" Oct 30 21:50:41.497523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.497402 6808 scope.go:122] "RemoveContainer" containerID="c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1" Oct 30 21:50:41.497918 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:41.497893 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1\": container with ID starting with c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1 not found: ID does not exist" containerID="c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1" Oct 30 21:50:41.498136 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.497927 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1"} err="failed to get container status \"c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1\": rpc error: code = NotFound desc = could not find container \"c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1\": container with ID starting with c44546ef3b8e97df2459e534c9adc7bfae320b8182fdb9d42d341c079724bde1 not found: ID does not exist" Oct 30 21:50:41.509715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.509688 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9"] Oct 30 21:50:41.521507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.521479 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1f725d-openshift-default-656f4cc85c-dlrb9"] Oct 30 21:50:41.661813 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.661766 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs"] Oct 30 21:50:41.662837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:41.662316 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" podUID="74161860-6929-41a8-92e7-69d171cf972b" containerName="maas-api" containerID="cri-o://5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0" gracePeriod=30 Oct 30 21:50:42.155292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.155015 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815fbaae-7bbc-4560-928b-269a8b34634f" path="/var/lib/kubelet/pods/815fbaae-7bbc-4560-928b-269a8b34634f/volumes" Oct 30 21:50:42.159549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.156036 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e60288-7a9a-449f-85c7-897f9b5a1b99" path="/var/lib/kubelet/pods/a2e60288-7a9a-449f-85c7-897f9b5a1b99/volumes" Oct 30 21:50:42.159549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.156729 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" path="/var/lib/kubelet/pods/cd168855-dfa6-412d-a18b-03a29a6bb7ca/volumes" Oct 30 21:50:42.242069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.242040 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:50:42.379797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.379759 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\") pod \"74161860-6929-41a8-92e7-69d171cf972b\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " Oct 30 21:50:42.379944 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.379887 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca\") pod \"74161860-6929-41a8-92e7-69d171cf972b\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " Oct 30 21:50:42.380029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.380016 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs\") pod \"74161860-6929-41a8-92e7-69d171cf972b\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " Oct 30 21:50:42.380127 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.380109 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2\" (UniqueName: \"kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2\") pod \"74161860-6929-41a8-92e7-69d171cf972b\" (UID: \"74161860-6929-41a8-92e7-69d171cf972b\") " Oct 30 21:50:42.380375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.380338 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca" pod "74161860-6929-41a8-92e7-69d171cf972b" (UID: "74161860-6929-41a8-92e7-69d171cf972b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:42.380763 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.380741 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74161860-6929-41a8-92e7-69d171cf972b-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:42.382068 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.382040 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls" pod "74161860-6929-41a8-92e7-69d171cf972b" (UID: "74161860-6929-41a8-92e7-69d171cf972b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:42.382199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.382174 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2" pod "74161860-6929-41a8-92e7-69d171cf972b" (UID: "74161860-6929-41a8-92e7-69d171cf972b"). InnerVolumeSpecName "kube-api-access-dz2z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:42.382326 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.382256 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs" pod "74161860-6929-41a8-92e7-69d171cf972b" (UID: "74161860-6929-41a8-92e7-69d171cf972b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:42.481331 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.481256 6808 generic.go:356] "Generic (PLEG): container finished" podID="74161860-6929-41a8-92e7-69d171cf972b" containerID="5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0" exitCode=0 Oct 30 21:50:42.482099 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.481302 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" event={"ID":"74161860-6929-41a8-92e7-69d171cf972b","Type":"ContainerDied","Data":"5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0"} Oct 30 21:50:42.482245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.482139 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" event={"ID":"74161860-6929-41a8-92e7-69d171cf972b","Type":"ContainerDied","Data":"206c4334a47df7ba2ad74a89633333f38f38793db5081a6ae03f288625e3ce21"} Oct 30 21:50:42.482245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.481764 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:42.482245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.481310 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" Oct 30 21:50:42.482245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.482183 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2z2\" (UniqueName: \"kubernetes.io/projected/74161860-6929-41a8-92e7-69d171cf972b-kube-api-access-dz2z2\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:42.482245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.482191 6808 scope.go:122] "RemoveContainer" containerID="5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0" Oct 30 21:50:42.482245 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.482199 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74161860-6929-41a8-92e7-69d171cf972b-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:42.493265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.493247 6808 scope.go:122] "RemoveContainer" containerID="5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0" Oct 30 21:50:42.493621 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:42.493600 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0\": container with ID starting with 5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0 not found: ID does not exist" containerID="5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0" Oct 30 21:50:42.493715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.493627 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0"} err="failed to get container status \"5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0\": rpc error: code = NotFound desc = could not find container \"5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0\": container with ID starting with 5d45333c477e0e4a6dc0753c5a54eec0e07ef75e204ceddbfc2eb2a5b4e670b0 not found: ID does not exist" Oct 30 21:50:42.508882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.508854 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs"] Oct 30 21:50:42.513561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:42.513540 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs"] Oct 30 21:50:43.928867 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:43.928834 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74161860-6929-41a8-92e7-69d171cf972b" path="/var/lib/kubelet/pods/74161860-6929-41a8-92e7-69d171cf972b/volumes" Oct 30 21:50:45.167009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:45.166966 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6cf5f0-78c49dbb4d-tfbqs" podUID="74161860-6929-41a8-92e7-69d171cf972b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.114:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:45.717012 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:45.716977 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" podUID="14e04609-0dd9-47e2-a05c-2d5762433b29" containerName="payload-processing" containerID="cri-o://9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8" gracePeriod=25 Oct 30 21:50:45.990909 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:45.990884 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:50:46.054886 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.054856 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj\" (UniqueName: \"kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj\") pod \"14e04609-0dd9-47e2-a05c-2d5762433b29\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " Oct 30 21:50:46.055083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.055023 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume\") pod \"14e04609-0dd9-47e2-a05c-2d5762433b29\" (UID: \"14e04609-0dd9-47e2-a05c-2d5762433b29\") " Oct 30 21:50:46.055449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.055416 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume" pod "14e04609-0dd9-47e2-a05c-2d5762433b29" (UID: "14e04609-0dd9-47e2-a05c-2d5762433b29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:46.055723 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.055702 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e04609-0dd9-47e2-a05c-2d5762433b29-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:46.057616 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.057582 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj" pod "14e04609-0dd9-47e2-a05c-2d5762433b29" (UID: "14e04609-0dd9-47e2-a05c-2d5762433b29"). InnerVolumeSpecName "kube-api-access-zqvlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:46.158320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.158282 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqvlj\" (UniqueName: \"kubernetes.io/projected/14e04609-0dd9-47e2-a05c-2d5762433b29-kube-api-access-zqvlj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:46.521821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.521789 6808 generic.go:356] "Generic (PLEG): container finished" podID="14e04609-0dd9-47e2-a05c-2d5762433b29" containerID="9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8" exitCode=0 Oct 30 21:50:46.522262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.521840 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" Oct 30 21:50:46.522262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.521882 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" event={"ID":"14e04609-0dd9-47e2-a05c-2d5762433b29","Type":"ContainerDied","Data":"9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8"} Oct 30 21:50:46.522262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.521974 6808 scope.go:122] "RemoveContainer" containerID="9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8" Oct 30 21:50:46.522262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.521944 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg" event={"ID":"14e04609-0dd9-47e2-a05c-2d5762433b29","Type":"ContainerDied","Data":"11e180f8b5f24f6397db21ffbe319465f467336dd66e926d48da4884757cf042"} Oct 30 21:50:46.532446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.532428 6808 scope.go:122] "RemoveContainer" containerID="9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8" Oct 30 21:50:46.532707 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:46.532690 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8\": container with ID starting with 9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8 not found: ID does not exist" containerID="9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8" Oct 30 21:50:46.532776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.532712 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8"} err="failed to get container status \"9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8\": rpc error: code = NotFound desc = could not find container \"9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8\": container with ID starting with 9d5157f5e8eac913d6b82b689034ed31a197cd8d5365c218f8bdfefecb12ffc8 not found: ID does not exist" Oct 30 21:50:46.547220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.547189 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg"] Oct 30 21:50:46.550443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:46.550423 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6cf5f0-7b96b6f785-bh5xg"] Oct 30 21:50:47.786090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:47.785915 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf"] Oct 30 21:50:47.796011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:47.786864 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" containerID="cri-o://b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b" gracePeriod=30 Oct 30 21:50:48.029972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:48.029330 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e04609-0dd9-47e2-a05c-2d5762433b29" path="/var/lib/kubelet/pods/14e04609-0dd9-47e2-a05c-2d5762433b29/volumes" Oct 30 21:50:51.309962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:51.309922 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2"] Oct 30 21:50:51.310447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:51.310210 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" containerID="cri-o://54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879" gracePeriod=30 Oct 30 21:50:53.075393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.075327 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:50:53.128186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128156 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128253 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128304 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128483 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128532 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128646 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128582 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128646 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128607 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.128745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128640 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:53.128745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128662 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.128745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128737 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128925 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128773 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.128925 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128828 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.128925 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.128908 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data\") pod \"5e659ef2-1952-4763-819a-f0034d970c84\" (UID: \"5e659ef2-1952-4763-819a-f0034d970c84\") " Oct 30 21:50:53.129222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.129199 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.129592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.129570 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659ef2-1952-4763-819a-f0034d970c84-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.129710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.129594 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.129710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.129609 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.129710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.129622 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.129710 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.129634 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.131288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.131261 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:53.131382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.131329 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:53.131446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.131425 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.134071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.134049 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch" pod "5e659ef2-1952-4763-819a-f0034d970c84" (UID: "5e659ef2-1952-4763-819a-f0034d970c84"). InnerVolumeSpecName "kube-api-access-r92ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:53.230793 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.230762 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92ch\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-kube-api-access-r92ch\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.230793 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.230791 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659ef2-1952-4763-819a-f0034d970c84-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.231022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.230805 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659ef2-1952-4763-819a-f0034d970c84-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.231022 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.230818 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659ef2-1952-4763-819a-f0034d970c84-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.674071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.674026 6808 generic.go:356] "Generic (PLEG): container finished" podID="5e659ef2-1952-4763-819a-f0034d970c84" containerID="b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b" exitCode=0 Oct 30 21:50:53.674267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.674242 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" event={"ID":"5e659ef2-1952-4763-819a-f0034d970c84","Type":"ContainerDied","Data":"b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b"} Oct 30 21:50:53.674342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.674293 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" event={"ID":"5e659ef2-1952-4763-819a-f0034d970c84","Type":"ContainerDied","Data":"08930681bd3a20ceebf8c12d95607b082d49bc590bb43379937e72219ae6f878"} Oct 30 21:50:53.674342 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.674320 6808 scope.go:122] "RemoveContainer" containerID="b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b" Oct 30 21:50:53.674423 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.674362 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf" Oct 30 21:50:53.688632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.688610 6808 scope.go:122] "RemoveContainer" containerID="883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be" Oct 30 21:50:53.699781 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.699764 6808 scope.go:122] "RemoveContainer" containerID="b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b" Oct 30 21:50:53.700097 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:53.700077 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b\": container with ID starting with b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b not found: ID does not exist" containerID="b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b" Oct 30 21:50:53.700209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.700106 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b"} err="failed to get container status \"b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b\": rpc error: code = NotFound desc = could not find container \"b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b\": container with ID starting with b40a6747514e77500fdda4bbb99dd6fb685fa35cd5c02a1ac531428b4cc3606b not found: ID does not exist" Oct 30 21:50:53.700209 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.700126 6808 scope.go:122] "RemoveContainer" containerID="883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be" Oct 30 21:50:53.700554 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:53.700522 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be\": container with ID starting with 883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be not found: ID does not exist" containerID="883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be" Oct 30 21:50:53.700664 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.700555 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be"} err="failed to get container status \"883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be\": rpc error: code = NotFound desc = could not find container \"883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be\": container with ID starting with 883ee7ca31c0858a665a834db1700ad116d76149b07de3f96901aadab70305be not found: ID does not exist" Oct 30 21:50:53.703021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.703001 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf"] Oct 30 21:50:53.712211 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.712191 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-2lkqf"] Oct 30 21:50:53.928746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:53.928672 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e659ef2-1952-4763-819a-f0034d970c84" path="/var/lib/kubelet/pods/5e659ef2-1952-4763-819a-f0034d970c84/volumes" Oct 30 21:50:56.037036 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.036997 6808 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:56.037413 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.037066 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:56.644436 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.644407 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:50:56.710313 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.710268 6808 generic.go:356] "Generic (PLEG): container finished" podID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerID="54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879" exitCode=0 Oct 30 21:50:56.710504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.710349 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" event={"ID":"219686ae-b68b-4d6d-b32a-0de8543b2064","Type":"ContainerDied","Data":"54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879"} Oct 30 21:50:56.710504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.710415 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" event={"ID":"219686ae-b68b-4d6d-b32a-0de8543b2064","Type":"ContainerDied","Data":"3ebf314287d73c6deb9d4c88fe4f0c6cd0b155ef4e8ff92ee8f0e104c0a5b4f3"} Oct 30 21:50:56.710504 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.710448 6808 scope.go:122] "RemoveContainer" containerID="54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879" Oct 30 21:50:56.710703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.710564 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2" Oct 30 21:50:56.716470 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716444 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.716634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716556 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.716634 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716613 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.716802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716703 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.716802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716759 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.716802 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716792 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.717007 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.716860 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.717089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717013 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.717159 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717123 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket\") pod \"219686ae-b68b-4d6d-b32a-0de8543b2064\" (UID: \"219686ae-b68b-4d6d-b32a-0de8543b2064\") " Oct 30 21:50:56.717222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717189 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.717222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717203 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.717707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717626 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.717707 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717675 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.717874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717796 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.717874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717815 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:56.717874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717818 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.717874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.717860 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.721337 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.721232 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:56.721337 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.721312 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.721626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.721600 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "kube-api-access-2hs2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:56.721911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.721840 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo" pod "219686ae-b68b-4d6d-b32a-0de8543b2064" (UID: "219686ae-b68b-4d6d-b32a-0de8543b2064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:56.722017 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.721986 6808 scope.go:122] "RemoveContainer" containerID="97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b" Oct 30 21:50:56.745632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.745607 6808 scope.go:122] "RemoveContainer" containerID="54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879" Oct 30 21:50:56.745979 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:56.745933 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879\": container with ID starting with 54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879 not found: ID does not exist" containerID="54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879" Oct 30 21:50:56.746051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.745976 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879"} err="failed to get container status \"54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879\": rpc error: code = NotFound desc = could not find container \"54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879\": container with ID starting with 54781fab307f7c142758cebd5169a1ab7b5346a94637a22a4e934c4ad896c879 not found: ID does not exist" Oct 30 21:50:56.746051 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.745996 6808 scope.go:122] "RemoveContainer" containerID="97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b" Oct 30 21:50:56.746290 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:50:56.746273 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b\": container with ID starting with 97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b not found: ID does not exist" containerID="97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b" Oct 30 21:50:56.746343 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.746294 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b"} err="failed to get container status \"97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b\": rpc error: code = NotFound desc = could not find container \"97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b\": container with ID starting with 97dd83bc19a53cbae21bb9f32af3ec9931a390a8ba993c0e2acd9652ddda727b not found: ID does not exist" Oct 30 21:50:56.819785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.819747 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.819785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.819783 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.819986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.819798 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219686ae-b68b-4d6d-b32a-0de8543b2064-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.819986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.819813 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.819986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.819827 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219686ae-b68b-4d6d-b32a-0de8543b2064-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.819986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:56.819842 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hs2q\" (UniqueName: \"kubernetes.io/projected/219686ae-b68b-4d6d-b32a-0de8543b2064-kube-api-access-2hs2q\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:57.045310 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:57.045281 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2"] Oct 30 21:50:57.057284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:57.057257 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2b36c1-openshift-default-577f5fbf67-b2hc2"] Oct 30 21:50:57.930165 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:50:57.930138 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" path="/var/lib/kubelet/pods/219686ae-b68b-4d6d-b32a-0de8543b2064/volumes" Oct 30 21:51:01.540667 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:01.540633 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf"] Oct 30 21:51:02.041971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:02.041194 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz"] Oct 30 21:51:02.048009 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:02.047979 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh"] Oct 30 21:51:04.911679 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:04.911592 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr"] Oct 30 21:51:04.912251 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:04.912051 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" podUID="c39d8c88-44e7-428a-9425-92218b68f58e" containerName="maas-api" containerID="cri-o://7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b" gracePeriod=30 Oct 30 21:51:05.214028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.191989 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k"] Oct 30 21:51:05.234977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.232108 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t"] Oct 30 21:51:05.612971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.607365 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:51:05.723058 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:05.722972 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:51:05.727585 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.727565 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls\") pod \"c39d8c88-44e7-428a-9425-92218b68f58e\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " Oct 30 21:51:05.727680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.727602 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs\") pod \"c39d8c88-44e7-428a-9425-92218b68f58e\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " Oct 30 21:51:05.727680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.727625 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca\") pod \"c39d8c88-44e7-428a-9425-92218b68f58e\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " Oct 30 21:51:05.727804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.727789 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t\" (UniqueName: \"kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t\") pod \"c39d8c88-44e7-428a-9425-92218b68f58e\" (UID: \"c39d8c88-44e7-428a-9425-92218b68f58e\") " Oct 30 21:51:05.728082 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.728057 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca" pod "c39d8c88-44e7-428a-9425-92218b68f58e" (UID: "c39d8c88-44e7-428a-9425-92218b68f58e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:05.728391 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.728373 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d8c88-44e7-428a-9425-92218b68f58e-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:05.729607 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.729582 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs" pod "c39d8c88-44e7-428a-9425-92218b68f58e" (UID: "c39d8c88-44e7-428a-9425-92218b68f58e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:05.729713 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.729661 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls" pod "c39d8c88-44e7-428a-9425-92218b68f58e" (UID: "c39d8c88-44e7-428a-9425-92218b68f58e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:05.729789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.729772 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t" pod "c39d8c88-44e7-428a-9425-92218b68f58e" (UID: "c39d8c88-44e7-428a-9425-92218b68f58e"). InnerVolumeSpecName "kube-api-access-5tj5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:05.781741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.781710 6808 generic.go:356] "Generic (PLEG): container finished" podID="c39d8c88-44e7-428a-9425-92218b68f58e" containerID="7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b" exitCode=0 Oct 30 21:51:05.782016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.781765 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" Oct 30 21:51:05.782016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.781836 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" event={"ID":"c39d8c88-44e7-428a-9425-92218b68f58e","Type":"ContainerDied","Data":"7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b"} Oct 30 21:51:05.782016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.781875 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr" event={"ID":"c39d8c88-44e7-428a-9425-92218b68f58e","Type":"ContainerDied","Data":"e9878f5b8003459a2d722567a2093dcef02caed999df7a5a2bf1adfade89742e"} Oct 30 21:51:05.782016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.781894 6808 scope.go:122] "RemoveContainer" containerID="7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b" Oct 30 21:51:05.794913 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.794890 6808 scope.go:122] "RemoveContainer" containerID="7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b" Oct 30 21:51:05.795292 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:05.795272 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b\": container with ID starting with 7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b not found: ID does not exist" containerID="7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b" Oct 30 21:51:05.795361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.795306 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b"} err="failed to get container status \"7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b\": rpc error: code = NotFound desc = could not find container \"7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b\": container with ID starting with 7bb08ec7854193e9d93af9ab9007aa8031fdc9b2537a3f31f60aea5e080cf24b not found: ID does not exist" Oct 30 21:51:05.807776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.807750 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr"] Oct 30 21:51:05.814750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.814661 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-dc3edf-587d8f5d47-spdnr"] Oct 30 21:51:05.830861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.830823 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:05.830861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.830855 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39d8c88-44e7-428a-9425-92218b68f58e-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:05.830861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.830868 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tj5t\" (UniqueName: \"kubernetes.io/projected/c39d8c88-44e7-428a-9425-92218b68f58e-kube-api-access-5tj5t\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:05.929594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:05.929546 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39d8c88-44e7-428a-9425-92218b68f58e" path="/var/lib/kubelet/pods/c39d8c88-44e7-428a-9425-92218b68f58e/volumes" Oct 30 21:51:06.545232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:06.545181 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" podUID="cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" containerName="payload-pre-processing" containerID="cri-o://1906f750cb8b19c59380a0f3400e4af23de446b663d0ac8452ca9450fe002e2e" gracePeriod=25 Oct 30 21:51:06.823880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:06.823850 6808 generic.go:356] "Generic (PLEG): container finished" podID="cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" containerID="1906f750cb8b19c59380a0f3400e4af23de446b663d0ac8452ca9450fe002e2e" exitCode=0 Oct 30 21:51:06.824081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:06.824062 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" event={"ID":"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3","Type":"ContainerDied","Data":"1906f750cb8b19c59380a0f3400e4af23de446b663d0ac8452ca9450fe002e2e"} Oct 30 21:51:07.060224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.059448 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" podUID="8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" containerName="payload-pre-processing" containerID="cri-o://2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe" gracePeriod=25 Oct 30 21:51:07.060224 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.059648 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" podUID="70f1c4ee-9b0d-4283-ac84-77d7f209eb14" containerName="payload-processing" containerID="cri-o://9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d" gracePeriod=25 Oct 30 21:51:07.091028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.088752 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:51:07.185148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.184122 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume\") pod \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " Oct 30 21:51:07.185148 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.184233 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz\" (UniqueName: \"kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz\") pod \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\" (UID: \"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3\") " Oct 30 21:51:07.185373 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.185187 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume" pod "cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" (UID: "cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:07.192075 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.192052 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz" pod "cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" (UID: "cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3"). InnerVolumeSpecName "kube-api-access-45fdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:07.298178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.298110 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.298178 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.298146 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45fdz\" (UniqueName: \"kubernetes.io/projected/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3-kube-api-access-45fdz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.403068 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.402924 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:51:07.475492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.475456 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:51:07.500994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.500924 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx\" (UniqueName: \"kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx\") pod \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " Oct 30 21:51:07.502052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.502021 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume\") pod \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\" (UID: \"70f1c4ee-9b0d-4283-ac84-77d7f209eb14\") " Oct 30 21:51:07.503152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.503128 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume" pod "70f1c4ee-9b0d-4283-ac84-77d7f209eb14" (UID: "70f1c4ee-9b0d-4283-ac84-77d7f209eb14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:07.504108 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.504085 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx" pod "70f1c4ee-9b0d-4283-ac84-77d7f209eb14" (UID: "70f1c4ee-9b0d-4283-ac84-77d7f209eb14"). InnerVolumeSpecName "kube-api-access-l8skx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:07.631566 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.631473 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc\" (UniqueName: \"kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc\") pod \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " Oct 30 21:51:07.631743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.631571 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume\") pod \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\" (UID: \"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5\") " Oct 30 21:51:07.632105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.632021 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume" pod "8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" (UID: "8cadc6b8-d1c3-4156-8970-ecfc72bebfc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:07.632888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.632860 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.633000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.632894 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.633000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.632910 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8skx\" (UniqueName: \"kubernetes.io/projected/70f1c4ee-9b0d-4283-ac84-77d7f209eb14-kube-api-access-l8skx\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.634750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.634725 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc" pod "8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" (UID: "8cadc6b8-d1c3-4156-8970-ecfc72bebfc5"). InnerVolumeSpecName "kube-api-access-6t4cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:07.734688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.734654 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4cc\" (UniqueName: \"kubernetes.io/projected/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5-kube-api-access-6t4cc\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.831897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.831864 6808 generic.go:356] "Generic (PLEG): container finished" podID="70f1c4ee-9b0d-4283-ac84-77d7f209eb14" containerID="9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d" exitCode=0 Oct 30 21:51:07.832096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.831903 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" Oct 30 21:51:07.832096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.831965 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" event={"ID":"70f1c4ee-9b0d-4283-ac84-77d7f209eb14","Type":"ContainerDied","Data":"9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d"} Oct 30 21:51:07.832096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.832010 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz" event={"ID":"70f1c4ee-9b0d-4283-ac84-77d7f209eb14","Type":"ContainerDied","Data":"dc302e489fa02c4b9cc21a27ccc39b0ebd84d391c90c8a01bad03d08b91cc06b"} Oct 30 21:51:07.832096 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.832032 6808 scope.go:122] "RemoveContainer" containerID="9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d" Oct 30 21:51:07.833385 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.833364 6808 generic.go:356] "Generic (PLEG): container finished" podID="8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" containerID="2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe" exitCode=0 Oct 30 21:51:07.833463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.833413 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" Oct 30 21:51:07.833463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.833445 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" event={"ID":"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5","Type":"ContainerDied","Data":"2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe"} Oct 30 21:51:07.833577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.833497 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh" event={"ID":"8cadc6b8-d1c3-4156-8970-ecfc72bebfc5","Type":"ContainerDied","Data":"f06b42ef56ff92ef471cd41f55fd2140c8752c0ddc225051f3cc2bb6d60dd399"} Oct 30 21:51:07.834997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.834972 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" event={"ID":"cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3","Type":"ContainerDied","Data":"e55148abf029b5d7147f27c183932fd0901d63fb1ae750bf984e6be6cbc99b74"} Oct 30 21:51:07.835116 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.834984 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf" Oct 30 21:51:07.879909 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.876078 6808 scope.go:122] "RemoveContainer" containerID="9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d" Oct 30 21:51:07.879909 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:07.877181 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d\": container with ID starting with 9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d not found: ID does not exist" containerID="9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d" Oct 30 21:51:07.879909 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.877215 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d"} err="failed to get container status \"9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d\": rpc error: code = NotFound desc = could not find container \"9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d\": container with ID starting with 9f1f330cfea0a05b23d90956010791a4bb35744b50e47897eeed48ae893f480d not found: ID does not exist" Oct 30 21:51:07.879909 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.877240 6808 scope.go:122] "RemoveContainer" containerID="2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe" Oct 30 21:51:07.890728 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.890710 6808 scope.go:122] "RemoveContainer" containerID="2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe" Oct 30 21:51:07.891046 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:07.891020 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe\": container with ID starting with 2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe not found: ID does not exist" containerID="2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe" Oct 30 21:51:07.891156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.891055 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe"} err="failed to get container status \"2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe\": rpc error: code = NotFound desc = could not find container \"2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe\": container with ID starting with 2639b4696e844045c27794cf454155ca05c6ffe09fe5fe6bef1c16c22be73bfe not found: ID does not exist" Oct 30 21:51:07.891156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.891076 6808 scope.go:122] "RemoveContainer" containerID="1906f750cb8b19c59380a0f3400e4af23de446b663d0ac8452ca9450fe002e2e" Oct 30 21:51:07.901259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.901236 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz"] Oct 30 21:51:07.905404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.905385 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-dc3edf-5c76cf98b7-ww9mz"] Oct 30 21:51:07.916135 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.916115 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh"] Oct 30 21:51:07.938939 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.938912 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f1c4ee-9b0d-4283-ac84-77d7f209eb14" path="/var/lib/kubelet/pods/70f1c4ee-9b0d-4283-ac84-77d7f209eb14/volumes" Oct 30 21:51:07.939301 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.939287 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dc3edf-86cfb6898c-nsdkh"] Oct 30 21:51:07.951979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.949912 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf"] Oct 30 21:51:07.975416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:07.975388 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-aad4dd-7d95fc69f-8jknf"] Oct 30 21:51:08.183024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:08.179175 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s"] Oct 30 21:51:08.183024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:08.179407 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" podUID="27aea3b1-9630-4c78-9ae5-10fbd1718ec0" containerName="maas-api" containerID="cri-o://bef514d5c9c8f9fc287d22ddef6aa158b9fbe03547eed36976c709939641feae" gracePeriod=30 Oct 30 21:51:08.929069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:08.929034 6808 generic.go:356] "Generic (PLEG): container finished" podID="27aea3b1-9630-4c78-9ae5-10fbd1718ec0" containerID="bef514d5c9c8f9fc287d22ddef6aa158b9fbe03547eed36976c709939641feae" exitCode=0 Oct 30 21:51:08.929284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:08.929193 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" event={"ID":"27aea3b1-9630-4c78-9ae5-10fbd1718ec0","Type":"ContainerDied","Data":"bef514d5c9c8f9fc287d22ddef6aa158b9fbe03547eed36976c709939641feae"} Oct 30 21:51:09.084353 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.084322 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:51:09.130967 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.130921 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5\") pod \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " Oct 30 21:51:09.131145 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.130999 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca\") pod \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " Oct 30 21:51:09.131145 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.131129 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls\") pod \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " Oct 30 21:51:09.131286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.131153 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs\") pod \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\" (UID: \"27aea3b1-9630-4c78-9ae5-10fbd1718ec0\") " Oct 30 21:51:09.132306 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.132283 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca" pod "27aea3b1-9630-4c78-9ae5-10fbd1718ec0" (UID: "27aea3b1-9630-4c78-9ae5-10fbd1718ec0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:09.133827 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.133804 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs" pod "27aea3b1-9630-4c78-9ae5-10fbd1718ec0" (UID: "27aea3b1-9630-4c78-9ae5-10fbd1718ec0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:09.133915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.133892 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls" pod "27aea3b1-9630-4c78-9ae5-10fbd1718ec0" (UID: "27aea3b1-9630-4c78-9ae5-10fbd1718ec0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:09.134530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.134504 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5" pod "27aea3b1-9630-4c78-9ae5-10fbd1718ec0" (UID: "27aea3b1-9630-4c78-9ae5-10fbd1718ec0"). InnerVolumeSpecName "kube-api-access-xfpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:09.201686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.201648 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24"] Oct 30 21:51:09.221432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.221404 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj"] Oct 30 21:51:09.232753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.232726 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpp5\" (UniqueName: \"kubernetes.io/projected/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-kube-api-access-xfpp5\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:09.232753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.232753 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:09.232965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.232768 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:09.232965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.232780 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aea3b1-9630-4c78-9ae5-10fbd1718ec0-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:09.930567 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.930533 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" path="/var/lib/kubelet/pods/8cadc6b8-d1c3-4156-8970-ecfc72bebfc5/volumes" Oct 30 21:51:09.931160 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.931134 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" path="/var/lib/kubelet/pods/cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3/volumes" Oct 30 21:51:09.935704 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.935677 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" Oct 30 21:51:09.935829 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.935710 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s" event={"ID":"27aea3b1-9630-4c78-9ae5-10fbd1718ec0","Type":"ContainerDied","Data":"b57a063aa79e4ab346ccd53401a9edc6007f99fa980442fd1c4858ed37cf728b"} Oct 30 21:51:09.935829 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:09.935767 6808 scope.go:122] "RemoveContainer" containerID="bef514d5c9c8f9fc287d22ddef6aa158b9fbe03547eed36976c709939641feae" Oct 30 21:51:10.002726 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.002691 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s"] Oct 30 21:51:10.009488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.009459 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0b412a-55c9554f4f-52m8s"] Oct 30 21:51:10.239468 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.239387 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" podUID="6017d829-5ae1-43ae-a65e-9bb1c856db13" containerName="payload-processing" containerID="cri-o://9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920" gracePeriod=25 Oct 30 21:51:10.239912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.239391 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" podUID="adbedada-9cf6-4d94-9edb-0f9f92010d5c" containerName="payload-pre-processing" containerID="cri-o://025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7" gracePeriod=25 Oct 30 21:51:10.717904 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.717880 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:51:10.790780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.790751 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:51:10.819910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.819834 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume\") pod \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " Oct 30 21:51:10.820098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.820013 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj\" (UniqueName: \"kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj\") pod \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\" (UID: \"adbedada-9cf6-4d94-9edb-0f9f92010d5c\") " Oct 30 21:51:10.820294 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.820265 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume" pod "adbedada-9cf6-4d94-9edb-0f9f92010d5c" (UID: "adbedada-9cf6-4d94-9edb-0f9f92010d5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:10.820660 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.820634 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbedada-9cf6-4d94-9edb-0f9f92010d5c-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.822986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.822922 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj" pod "adbedada-9cf6-4d94-9edb-0f9f92010d5c" (UID: "adbedada-9cf6-4d94-9edb-0f9f92010d5c"). InnerVolumeSpecName "kube-api-access-glkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:10.921863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.921822 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8\" (UniqueName: \"kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8\") pod \"6017d829-5ae1-43ae-a65e-9bb1c856db13\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " Oct 30 21:51:10.922083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.921889 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume\") pod \"6017d829-5ae1-43ae-a65e-9bb1c856db13\" (UID: \"6017d829-5ae1-43ae-a65e-9bb1c856db13\") " Oct 30 21:51:10.922265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.922250 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkhj\" (UniqueName: \"kubernetes.io/projected/adbedada-9cf6-4d94-9edb-0f9f92010d5c-kube-api-access-glkhj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.922308 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.922270 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume" pod "6017d829-5ae1-43ae-a65e-9bb1c856db13" (UID: "6017d829-5ae1-43ae-a65e-9bb1c856db13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:10.924316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.924277 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8" pod "6017d829-5ae1-43ae-a65e-9bb1c856db13" (UID: "6017d829-5ae1-43ae-a65e-9bb1c856db13"). InnerVolumeSpecName "kube-api-access-9jxv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:10.995557 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.995506 6808 generic.go:356] "Generic (PLEG): container finished" podID="6017d829-5ae1-43ae-a65e-9bb1c856db13" containerID="9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920" exitCode=0 Oct 30 21:51:10.995815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.995671 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" event={"ID":"6017d829-5ae1-43ae-a65e-9bb1c856db13","Type":"ContainerDied","Data":"9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920"} Oct 30 21:51:10.995815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.995710 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" event={"ID":"6017d829-5ae1-43ae-a65e-9bb1c856db13","Type":"ContainerDied","Data":"deaa4ee989e40f35a3d16e7bd667b4c06d0cdb0a3a73b9b79371312b82e53760"} Oct 30 21:51:10.995815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.995732 6808 scope.go:122] "RemoveContainer" containerID="9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920" Oct 30 21:51:10.996159 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.996124 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k" Oct 30 21:51:10.999581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.999557 6808 generic.go:356] "Generic (PLEG): container finished" podID="adbedada-9cf6-4d94-9edb-0f9f92010d5c" containerID="025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7" exitCode=0 Oct 30 21:51:10.999738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.999628 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" event={"ID":"adbedada-9cf6-4d94-9edb-0f9f92010d5c","Type":"ContainerDied","Data":"025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7"} Oct 30 21:51:10.999887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.999858 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" event={"ID":"adbedada-9cf6-4d94-9edb-0f9f92010d5c","Type":"ContainerDied","Data":"dd177d6d9b8b2286a90e281f26a42c225801fa96509244b6bc43ec896175987c"} Oct 30 21:51:10.999887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:10.999656 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t" Oct 30 21:51:11.026700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.022228 6808 scope.go:122] "RemoveContainer" containerID="9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920" Oct 30 21:51:11.026700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.024018 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxv8\" (UniqueName: \"kubernetes.io/projected/6017d829-5ae1-43ae-a65e-9bb1c856db13-kube-api-access-9jxv8\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:11.026700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.024052 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017d829-5ae1-43ae-a65e-9bb1c856db13-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:11.030236 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:11.027345 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920\": container with ID starting with 9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920 not found: ID does not exist" containerID="9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920" Oct 30 21:51:11.030236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.027386 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920"} err="failed to get container status \"9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920\": rpc error: code = NotFound desc = could not find container \"9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920\": container with ID starting with 9ef55397d0b7d7df30d68f08419e6b262b0abe1ee9aa66141067a20f4c715920 not found: ID does not exist" Oct 30 21:51:11.030236 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.027411 6808 scope.go:122] "RemoveContainer" containerID="025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7" Oct 30 21:51:11.163188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.153778 6808 scope.go:122] "RemoveContainer" containerID="025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7" Oct 30 21:51:11.163188 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:11.154302 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7\": container with ID starting with 025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7 not found: ID does not exist" containerID="025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7" Oct 30 21:51:11.163188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.154338 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7"} err="failed to get container status \"025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7\": rpc error: code = NotFound desc = could not find container \"025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7\": container with ID starting with 025652d433ea9d1a3a690b9a1700195421e7c4d7e48c5ea1b12862ce086034b7 not found: ID does not exist" Oct 30 21:51:11.169987 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.163878 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t"] Oct 30 21:51:11.177080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.176453 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0b412a-56f7d4cb99-mhc4t"] Oct 30 21:51:11.267059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.264062 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k"] Oct 30 21:51:11.316728 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.316691 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-0b412a-66d68874dc-lh76k"] Oct 30 21:51:11.928748 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.928715 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27aea3b1-9630-4c78-9ae5-10fbd1718ec0" path="/var/lib/kubelet/pods/27aea3b1-9630-4c78-9ae5-10fbd1718ec0/volumes" Oct 30 21:51:11.929152 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.929138 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6017d829-5ae1-43ae-a65e-9bb1c856db13" path="/var/lib/kubelet/pods/6017d829-5ae1-43ae-a65e-9bb1c856db13/volumes" Oct 30 21:51:11.929532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:11.929519 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbedada-9cf6-4d94-9edb-0f9f92010d5c" path="/var/lib/kubelet/pods/adbedada-9cf6-4d94-9edb-0f9f92010d5c/volumes" Oct 30 21:51:14.616980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:14.615052 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2"] Oct 30 21:51:16.924518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:16.924482 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx"] Oct 30 21:51:16.925082 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:16.924781 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" podUID="6bb6ce41-4a25-40f7-8a38-bdfb625c9653" containerName="istio-proxy" containerID="cri-o://3808aa0bc3d082129ed61048ccff039d8d91e8f946cc1180ac20340ccf13477b" gracePeriod=30 Oct 30 21:51:17.144986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.144806 6808 generic.go:356] "Generic (PLEG): container finished" podID="6bb6ce41-4a25-40f7-8a38-bdfb625c9653" containerID="3808aa0bc3d082129ed61048ccff039d8d91e8f946cc1180ac20340ccf13477b" exitCode=0 Oct 30 21:51:17.145201 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.145004 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" event={"ID":"6bb6ce41-4a25-40f7-8a38-bdfb625c9653","Type":"ContainerDied","Data":"3808aa0bc3d082129ed61048ccff039d8d91e8f946cc1180ac20340ccf13477b"} Oct 30 21:51:17.268219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.268193 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:51:17.334600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.334552 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.356084 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.356047 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.361613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.356863 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.381016 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.356729 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.381327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.381303 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.381839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.381495 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.382055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.381382 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.382189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.381811 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.382392 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.382372 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.382552 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.382536 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.382777 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.382760 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.382921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.382907 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo\") pod \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\" (UID: \"6bb6ce41-4a25-40f7-8a38-bdfb625c9653\") " Oct 30 21:51:17.385686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.385653 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.385834 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.385818 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.385931 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.385919 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.390031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.389705 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.390031 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.389996 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:17.390364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.390331 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.394192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.394167 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:17.398446 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.398421 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "kube-api-access-fgkk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:17.400163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.400130 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token" pod "6bb6ce41-4a25-40f7-8a38-bdfb625c9653" (UID: "6bb6ce41-4a25-40f7-8a38-bdfb625c9653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:17.487214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.487144 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgkk7\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-kube-api-access-fgkk7\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.487214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.487177 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.487214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.487192 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.487214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.487207 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.487516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.487220 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.487516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.487235 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bb6ce41-4a25-40f7-8a38-bdfb625c9653-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.945632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.945600 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h"] Oct 30 21:51:17.946204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946186 6808 cpu_manager.go:414] "containerMap: removing container" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946206 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946221 6808 cpu_manager.go:414] "containerMap: removing container" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946226 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946232 6808 cpu_manager.go:414] "containerMap: removing container" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946238 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946250 6808 cpu_manager.go:414] "containerMap: removing container" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946254 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946265 6808 cpu_manager.go:414] "containerMap: removing container" podUID="27aea3b1-9630-4c78-9ae5-10fbd1718ec0" containerName="maas-api" Oct 30 21:51:17.946265 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946269 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27aea3b1-9630-4c78-9ae5-10fbd1718ec0" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946275 6808 cpu_manager.go:414] "containerMap: removing container" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946280 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946290 6808 cpu_manager.go:414] "containerMap: removing container" podUID="815fbaae-7bbc-4560-928b-269a8b34634f" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946294 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815fbaae-7bbc-4560-928b-269a8b34634f" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946300 6808 cpu_manager.go:414] "containerMap: removing container" podUID="9b7dfc83-82fe-43b7-af39-bcad0c9a7957" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946304 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7dfc83-82fe-43b7-af39-bcad0c9a7957" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946311 6808 cpu_manager.go:414] "containerMap: removing container" podUID="bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946315 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946323 6808 cpu_manager.go:414] "containerMap: removing container" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946327 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946334 6808 cpu_manager.go:414] "containerMap: removing container" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946338 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946346 6808 cpu_manager.go:414] "containerMap: removing container" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946350 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946356 6808 cpu_manager.go:414] "containerMap: removing container" podUID="14e04609-0dd9-47e2-a05c-2d5762433b29" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946360 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e04609-0dd9-47e2-a05c-2d5762433b29" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946367 6808 cpu_manager.go:414] "containerMap: removing container" podUID="2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946370 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946376 6808 cpu_manager.go:414] "containerMap: removing container" podUID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946380 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946387 6808 cpu_manager.go:414] "containerMap: removing container" podUID="c39d8c88-44e7-428a-9425-92218b68f58e" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946392 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39d8c88-44e7-428a-9425-92218b68f58e" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946398 6808 cpu_manager.go:414] "containerMap: removing container" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946403 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946409 6808 cpu_manager.go:414] "containerMap: removing container" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946413 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946420 6808 cpu_manager.go:414] "containerMap: removing container" podUID="adbedada-9cf6-4d94-9edb-0f9f92010d5c" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946424 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbedada-9cf6-4d94-9edb-0f9f92010d5c" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946431 6808 cpu_manager.go:414] "containerMap: removing container" podUID="70f1c4ee-9b0d-4283-ac84-77d7f209eb14" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946436 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f1c4ee-9b0d-4283-ac84-77d7f209eb14" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946444 6808 cpu_manager.go:414] "containerMap: removing container" podUID="cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946449 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946456 6808 cpu_manager.go:414] "containerMap: removing container" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946460 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946466 6808 cpu_manager.go:414] "containerMap: removing container" podUID="6bb6ce41-4a25-40f7-8a38-bdfb625c9653" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946470 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb6ce41-4a25-40f7-8a38-bdfb625c9653" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946479 6808 cpu_manager.go:414] "containerMap: removing container" podUID="74161860-6929-41a8-92e7-69d171cf972b" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946485 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74161860-6929-41a8-92e7-69d171cf972b" containerName="maas-api" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946497 6808 cpu_manager.go:414] "containerMap: removing container" podUID="8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946502 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946512 6808 cpu_manager.go:414] "containerMap: removing container" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946516 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946524 6808 cpu_manager.go:414] "containerMap: removing container" podUID="6017d829-5ae1-43ae-a65e-9bb1c856db13" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946528 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6017d829-5ae1-43ae-a65e-9bb1c856db13" containerName="payload-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946543 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a2e60288-7a9a-449f-85c7-897f9b5a1b99" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946548 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e60288-7a9a-449f-85c7-897f9b5a1b99" containerName="payload-pre-processing" Oct 30 21:51:17.946534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946556 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946561 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946633 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf2d7da-0533-4ea8-ae0e-1de61082c5a6" containerName="maas-api" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946643 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="8adc6e05-55c3-4d6c-9fc6-4bb2567d0b57" containerName="maas-api" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946653 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39d8c88-44e7-428a-9425-92218b68f58e" containerName="maas-api" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946664 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946671 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="4177f737-6bc7-4647-b1d8-351e85c33cee" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946681 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbedada-9cf6-4d94-9edb-0f9f92010d5c" containerName="payload-pre-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946689 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f1c4ee-9b0d-4283-ac84-77d7f209eb14" containerName="payload-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946700 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd6ff63-fbfe-4ad7-9a5e-f667aadd88f3" containerName="payload-pre-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946710 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="d367b0db-fb5d-4255-ac52-7a6c07d48275" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946718 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb6ce41-4a25-40f7-8a38-bdfb625c9653" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946726 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="74161860-6929-41a8-92e7-69d171cf972b" containerName="maas-api" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946736 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cadc6b8-d1c3-4156-8970-ecfc72bebfc5" containerName="payload-pre-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946747 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="348cec2d-fbc4-4adb-b8f6-2e50d4e1aeab" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946755 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="6017d829-5ae1-43ae-a65e-9bb1c856db13" containerName="payload-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946766 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e60288-7a9a-449f-85c7-897f9b5a1b99" containerName="payload-pre-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946772 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946780 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc2476c-7f34-4ed3-b880-b797ee00aa02" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946788 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946798 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b88d721-c32d-487a-b10d-1a5c48e940ce" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946806 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd168855-dfa6-412d-a18b-03a29a6bb7ca" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946814 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946821 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="27aea3b1-9630-4c78-9ae5-10fbd1718ec0" containerName="maas-api" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946829 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="219686ae-b68b-4d6d-b32a-0de8543b2064" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946838 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="815fbaae-7bbc-4560-928b-269a8b34634f" containerName="payload-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946845 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7dfc83-82fe-43b7-af39-bcad0c9a7957" containerName="payload-pre-processing" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946853 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd168fc-9a7b-4dbc-801a-a30cf715fdbc" containerName="maas-api" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946864 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946870 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="4724778b-c4f9-4b2e-9f4d-e08933ea4a16" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946878 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c75041-02c6-4326-ae01-bb651acf4497" containerName="istio-proxy" Oct 30 21:51:17.948682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.946886 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e04609-0dd9-47e2-a05c-2d5762433b29" containerName="payload-processing" Oct 30 21:51:17.954719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.954695 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:17.961848 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.961824 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:17.962006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.961890 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:17.963661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:17.963638 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h"] Oct 30 21:51:18.095404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.095365 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.095583 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.095543 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.095749 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.095713 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2dn\" (UniqueName: \"kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.154142 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.154118 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" Oct 30 21:51:18.154285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.154109 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx" event={"ID":"6bb6ce41-4a25-40f7-8a38-bdfb625c9653","Type":"ContainerDied","Data":"dc25d59cf50b3ae4612783fd2dea0daf67b10f4382ddcbacbf157fa66f300fa6"} Oct 30 21:51:18.154285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.154252 6808 scope.go:122] "RemoveContainer" containerID="3808aa0bc3d082129ed61048ccff039d8d91e8f946cc1180ac20340ccf13477b" Oct 30 21:51:18.177354 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.177328 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx"] Oct 30 21:51:18.185458 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.185433 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0b412a-openshift-default-74fc9f97f7-7p6sx"] Oct 30 21:51:18.196409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.196353 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2dn\" (UniqueName: \"kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.196517 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.196480 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.196568 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.196547 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.197243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.197220 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.198842 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.198811 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.209122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.209099 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2dn\" (UniqueName: \"kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn\") pod \"maas-api-revoke-sub-d0fca89e-bwv8h\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.295284 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.286290 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:18.635580 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:18.635556 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h"] Oct 30 21:51:18.637194 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:51:18.637169 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21be50d6_a832_443b_9e2f_f1e7f4f403b3.slice/crio-45e000c8bfaa8292741135b31584da68d4df06d65c1ea4a883645028e968dff0 WatchSource:0}: Error finding container 45e000c8bfaa8292741135b31584da68d4df06d65c1ea4a883645028e968dff0: Status 404 returned error can't find the container with id 45e000c8bfaa8292741135b31584da68d4df06d65c1ea4a883645028e968dff0 Oct 30 21:51:19.161921 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.161884 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" event={"ID":"21be50d6-a832-443b-9e2f-f1e7f4f403b3","Type":"ContainerStarted","Data":"dc27e4baf255c4b2579ec5cee2ab22368cca3d79c18c8f699a6b0173ea854059"} Oct 30 21:51:19.162400 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.161943 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" event={"ID":"21be50d6-a832-443b-9e2f-f1e7f4f403b3","Type":"ContainerStarted","Data":"45e000c8bfaa8292741135b31584da68d4df06d65c1ea4a883645028e968dff0"} Oct 30 21:51:19.184026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.183980 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" podStartSLOduration=2.183966769 podStartE2EDuration="2.183966769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.181647059 +0000 UTC m=+1341.377338179" watchObservedRunningTime="2026-10-30 21:51:19.183966769 +0000 UTC m=+1341.379657882" Oct 30 21:51:19.697804 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.694066 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerName="payload-pre-processing" containerID="cri-o://c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a" gracePeriod=25 Oct 30 21:51:19.928598 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.928567 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb6ce41-4a25-40f7-8a38-bdfb625c9653" path="/var/lib/kubelet/pods/6bb6ce41-4a25-40f7-8a38-bdfb625c9653/volumes" Oct 30 21:51:19.968962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.968920 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4"] Oct 30 21:51:19.993893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:19.993864 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc"] Oct 30 21:51:20.172831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:20.172794 6808 generic.go:356] "Generic (PLEG): container finished" podID="21be50d6-a832-443b-9e2f-f1e7f4f403b3" containerID="dc27e4baf255c4b2579ec5cee2ab22368cca3d79c18c8f699a6b0173ea854059" exitCode=0 Oct 30 21:51:20.173261 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:20.172879 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" event={"ID":"21be50d6-a832-443b-9e2f-f1e7f4f403b3","Type":"ContainerDied","Data":"dc27e4baf255c4b2579ec5cee2ab22368cca3d79c18c8f699a6b0173ea854059"} Oct 30 21:51:21.239555 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.239523 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz"] Oct 30 21:51:21.337807 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.337784 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:21.371923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.371885 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn\" (UniqueName: \"kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn\") pod \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " Oct 30 21:51:21.377687 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.371994 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca\") pod \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " Oct 30 21:51:21.377687 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.372154 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca\") pod \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\" (UID: \"21be50d6-a832-443b-9e2f-f1e7f4f403b3\") " Oct 30 21:51:21.377687 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.374106 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca" pod "21be50d6-a832-443b-9e2f-f1e7f4f403b3" (UID: "21be50d6-a832-443b-9e2f-f1e7f4f403b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:21.399640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.399601 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn" pod "21be50d6-a832-443b-9e2f-f1e7f4f403b3" (UID: "21be50d6-a832-443b-9e2f-f1e7f4f403b3"). InnerVolumeSpecName "kube-api-access-nn2dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:21.399640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.399603 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca" pod "21be50d6-a832-443b-9e2f-f1e7f4f403b3" (UID: "21be50d6-a832-443b-9e2f-f1e7f4f403b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:21.474640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.474545 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn2dn\" (UniqueName: \"kubernetes.io/projected/21be50d6-a832-443b-9e2f-f1e7f4f403b3-kube-api-access-nn2dn\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.474640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.474589 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.474640 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:21.474608 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21be50d6-a832-443b-9e2f-f1e7f4f403b3-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.196397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:22.196311 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" event={"ID":"21be50d6-a832-443b-9e2f-f1e7f4f403b3","Type":"ContainerDied","Data":"45e000c8bfaa8292741135b31584da68d4df06d65c1ea4a883645028e968dff0"} Oct 30 21:51:22.196397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:22.196360 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e000c8bfaa8292741135b31584da68d4df06d65c1ea4a883645028e968dff0" Oct 30 21:51:22.196397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:22.196328 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h" Oct 30 21:51:24.224558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.224521 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" containerID="cri-o://8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0" gracePeriod=45 Oct 30 21:51:24.274180 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.274145 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" containerID="cri-o://d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae" gracePeriod=45 Oct 30 21:51:24.343789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.343752 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Oct 30 21:51:24.569892 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.567440 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:51:24.617600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.617558 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:51:24.626463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626423 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir\") pod \"39bc077a-5dc9-419d-8f30-94cb2ea313da\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " Oct 30 21:51:24.626695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626502 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home\") pod \"39bc077a-5dc9-419d-8f30-94cb2ea313da\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " Oct 30 21:51:24.626695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626643 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache\") pod \"39bc077a-5dc9-419d-8f30-94cb2ea313da\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " Oct 30 21:51:24.626852 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626781 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz\" (UniqueName: \"kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz\") pod \"39bc077a-5dc9-419d-8f30-94cb2ea313da\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " Oct 30 21:51:24.626852 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626819 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm\") pod \"39bc077a-5dc9-419d-8f30-94cb2ea313da\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " Oct 30 21:51:24.626977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626902 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs\" (UniqueName: \"kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs\") pod \"39bc077a-5dc9-419d-8f30-94cb2ea313da\" (UID: \"39bc077a-5dc9-419d-8f30-94cb2ea313da\") " Oct 30 21:51:24.626977 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.626935 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache" pod "39bc077a-5dc9-419d-8f30-94cb2ea313da" (UID: "39bc077a-5dc9-419d-8f30-94cb2ea313da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.627443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.627415 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.627600 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.627575 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir" pod "39bc077a-5dc9-419d-8f30-94cb2ea313da" (UID: "39bc077a-5dc9-419d-8f30-94cb2ea313da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.627700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.627641 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home" pod "39bc077a-5dc9-419d-8f30-94cb2ea313da" (UID: "39bc077a-5dc9-419d-8f30-94cb2ea313da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.630131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.630106 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz" pod "39bc077a-5dc9-419d-8f30-94cb2ea313da" (UID: "39bc077a-5dc9-419d-8f30-94cb2ea313da"). InnerVolumeSpecName "kube-api-access-slmbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:24.630500 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.630476 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs" pod "39bc077a-5dc9-419d-8f30-94cb2ea313da" (UID: "39bc077a-5dc9-419d-8f30-94cb2ea313da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:24.631554 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.631526 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm" pod "39bc077a-5dc9-419d-8f30-94cb2ea313da" (UID: "39bc077a-5dc9-419d-8f30-94cb2ea313da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.727899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.727870 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm\") pod \"e9540531-1013-451b-b259-6fcf2ad094fb\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " Oct 30 21:51:24.728066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.727977 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache\") pod \"e9540531-1013-451b-b259-6fcf2ad094fb\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " Oct 30 21:51:24.728138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728122 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs\") pod \"e9540531-1013-451b-b259-6fcf2ad094fb\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " Oct 30 21:51:24.728223 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728200 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir\") pod \"e9540531-1013-451b-b259-6fcf2ad094fb\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " Oct 30 21:51:24.728335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728235 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache" pod "e9540531-1013-451b-b259-6fcf2ad094fb" (UID: "e9540531-1013-451b-b259-6fcf2ad094fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.728335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728272 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home\") pod \"e9540531-1013-451b-b259-6fcf2ad094fb\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " Oct 30 21:51:24.728435 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728337 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp\" (UniqueName: \"kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp\") pod \"e9540531-1013-451b-b259-6fcf2ad094fb\" (UID: \"e9540531-1013-451b-b259-6fcf2ad094fb\") " Oct 30 21:51:24.728492 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728467 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir" pod "e9540531-1013-451b-b259-6fcf2ad094fb" (UID: "e9540531-1013-451b-b259-6fcf2ad094fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.728579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728558 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home" pod "e9540531-1013-451b-b259-6fcf2ad094fb" (UID: "e9540531-1013-451b-b259-6fcf2ad094fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.728865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728846 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.728865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728867 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slmbz\" (UniqueName: \"kubernetes.io/projected/39bc077a-5dc9-419d-8f30-94cb2ea313da-kube-api-access-slmbz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728876 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728887 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bc077a-5dc9-419d-8f30-94cb2ea313da-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728896 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728904 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728911 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bc077a-5dc9-419d-8f30-94cb2ea313da-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.728918 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.729914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.729884 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm" pod "e9540531-1013-451b-b259-6fcf2ad094fb" (UID: "e9540531-1013-451b-b259-6fcf2ad094fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.730166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.730148 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs" pod "e9540531-1013-451b-b259-6fcf2ad094fb" (UID: "e9540531-1013-451b-b259-6fcf2ad094fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:24.730357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.730343 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp" pod "e9540531-1013-451b-b259-6fcf2ad094fb" (UID: "e9540531-1013-451b-b259-6fcf2ad094fb"). InnerVolumeSpecName "kube-api-access-7pksp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:24.830547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.830472 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pksp\" (UniqueName: \"kubernetes.io/projected/e9540531-1013-451b-b259-6fcf2ad094fb-kube-api-access-7pksp\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.830547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.830498 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9540531-1013-451b-b259-6fcf2ad094fb-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.830547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:24.830508 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9540531-1013-451b-b259-6fcf2ad094fb-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.217853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.217817 6808 generic.go:356] "Generic (PLEG): container finished" podID="e9540531-1013-451b-b259-6fcf2ad094fb" containerID="d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae" exitCode=0 Oct 30 21:51:25.218048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.217868 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" Oct 30 21:51:25.218048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.217905 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" event={"ID":"e9540531-1013-451b-b259-6fcf2ad094fb","Type":"ContainerDied","Data":"d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae"} Oct 30 21:51:25.218048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.217965 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj" event={"ID":"e9540531-1013-451b-b259-6fcf2ad094fb","Type":"ContainerDied","Data":"1431c4163844812bf5ab6e61fd815b8a960a52e8c9349a827e45646409716c6b"} Oct 30 21:51:25.218048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.217987 6808 scope.go:122] "RemoveContainer" containerID="d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae" Oct 30 21:51:25.219488 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.219464 6808 generic.go:356] "Generic (PLEG): container finished" podID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerID="8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0" exitCode=0 Oct 30 21:51:25.219601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.219503 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" Oct 30 21:51:25.219601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.219504 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" event={"ID":"39bc077a-5dc9-419d-8f30-94cb2ea313da","Type":"ContainerDied","Data":"8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0"} Oct 30 21:51:25.219601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.219539 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24" event={"ID":"39bc077a-5dc9-419d-8f30-94cb2ea313da","Type":"ContainerDied","Data":"79619c8150b52af99c492e477672cc2c02215f37cd8614b3588e3941a91f29fa"} Oct 30 21:51:25.229218 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.228943 6808 scope.go:122] "RemoveContainer" containerID="d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae" Oct 30 21:51:25.229794 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:25.229659 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae\": container with ID starting with d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae not found: ID does not exist" containerID="d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae" Oct 30 21:51:25.229794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.229682 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae"} err="failed to get container status \"d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae\": rpc error: code = NotFound desc = could not find container \"d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae\": container with ID starting with d878837491510d2415fd2c0153339cf0959dba0aecac1a75b3f84f70e07514ae not found: ID does not exist" Oct 30 21:51:25.229794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.229700 6808 scope.go:122] "RemoveContainer" containerID="8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0" Oct 30 21:51:25.246158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.246136 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj"] Oct 30 21:51:25.251207 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.251184 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/simulator-w2-db0b54-kserve-856f455cd6-jvgcj"] Oct 30 21:51:25.254114 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.254094 6808 scope.go:122] "RemoveContainer" containerID="8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0" Oct 30 21:51:25.254465 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:25.254445 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0\": container with ID starting with 8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0 not found: ID does not exist" containerID="8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0" Oct 30 21:51:25.254525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.254472 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0"} err="failed to get container status \"8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0\": rpc error: code = NotFound desc = could not find container \"8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0\": container with ID starting with 8ff1a1a62945f1a441d5daa7eed37f4cc9d4cfbd79fec4c94596f3fa130060f0 not found: ID does not exist" Oct 30 21:51:25.262280 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.262256 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24"] Oct 30 21:51:25.269712 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.269689 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-w2-db0b54-kserve-89c8db878-d7r24"] Oct 30 21:51:25.304292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.304267 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv"] Oct 30 21:51:25.959840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.959801 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" path="/var/lib/kubelet/pods/39bc077a-5dc9-419d-8f30-94cb2ea313da/volumes" Oct 30 21:51:25.960633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:25.960607 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" path="/var/lib/kubelet/pods/e9540531-1013-451b-b259-6fcf2ad094fb/volumes" Oct 30 21:51:26.280743 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:26.280652 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:51:28.291006 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.290970 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6"] Oct 30 21:51:28.291715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291693 6808 cpu_manager.go:414] "containerMap: removing container" podUID="21be50d6-a832-443b-9e2f-f1e7f4f403b3" containerName="revoke-keys" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291717 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21be50d6-a832-443b-9e2f-f1e7f4f403b3" containerName="revoke-keys" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291769 6808 cpu_manager.go:414] "containerMap: removing container" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291777 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291798 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291805 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291821 6808 cpu_manager.go:414] "containerMap: removing container" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" Oct 30 21:51:28.291861 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291829 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e659ef2-1952-4763-819a-f0034d970c84" containerName="istio-proxy" Oct 30 21:51:28.292150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291983 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bc077a-5dc9-419d-8f30-94cb2ea313da" containerName="main" Oct 30 21:51:28.292150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.291999 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9540531-1013-451b-b259-6fcf2ad094fb" containerName="main" Oct 30 21:51:28.292150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.292018 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="21be50d6-a832-443b-9e2f-f1e7f4f403b3" containerName="revoke-keys" Oct 30 21:51:28.297237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.297211 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.302225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.302174 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:28.303397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.303365 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:28.304592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.304558 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6"] Oct 30 21:51:28.470027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.469934 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvllj\" (UniqueName: \"kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.470355 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.470321 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.470439 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.470390 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.572444 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.572354 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvllj\" (UniqueName: \"kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.572613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.572472 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.572613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.572504 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.573386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.573358 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.575188 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.575161 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.585189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.585166 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvllj\" (UniqueName: \"kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj\") pod \"maas-api-revoke-sub-df361530-bdvz6\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.640970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.640172 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:28.964701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:28.964629 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6"] Oct 30 21:51:28.967962 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:51:28.967911 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166c9225_d920_4f04_bd04_d043b1228bd0.slice/crio-ee581eb34443e4f1bc28437ff283ed35023b6b1e42b5529862b87bfa5f3da071 WatchSource:0}: Error finding container ee581eb34443e4f1bc28437ff283ed35023b6b1e42b5529862b87bfa5f3da071: Status 404 returned error can't find the container with id ee581eb34443e4f1bc28437ff283ed35023b6b1e42b5529862b87bfa5f3da071 Oct 30 21:51:29.247629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:29.247593 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" event={"ID":"166c9225-d920-4f04-bd04-d043b1228bd0","Type":"ContainerStarted","Data":"ee581eb34443e4f1bc28437ff283ed35023b6b1e42b5529862b87bfa5f3da071"} Oct 30 21:51:30.255633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:30.255543 6808 generic.go:356] "Generic (PLEG): container finished" podID="166c9225-d920-4f04-bd04-d043b1228bd0" containerID="c3c66a4ca5a12ed8bc4450ce98d959649f61019de89851debcd1b0542dc0e69a" exitCode=0 Oct 30 21:51:30.256048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:30.255627 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" event={"ID":"166c9225-d920-4f04-bd04-d043b1228bd0","Type":"ContainerDied","Data":"c3c66a4ca5a12ed8bc4450ce98d959649f61019de89851debcd1b0542dc0e69a"} Oct 30 21:51:30.305403 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:30.305366 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" podUID="d1458c21-ef04-42e2-a973-dff27829548f" containerName="payload-pre-processing" containerID="cri-o://9dd3947b1c865d2701183ef84ffcc564521fb869468d0ccf552db9a83a68fbd1" gracePeriod=25 Oct 30 21:51:30.620129 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:30.620029 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp"] Oct 30 21:51:30.620328 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:30.620298 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" podUID="5261f014-d469-4320-b521-a15cbc87ce02" containerName="maas-api" containerID="cri-o://e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7" gracePeriod=30 Oct 30 21:51:31.149501 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.149472 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:51:31.275973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.272321 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7\" (UniqueName: \"kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7\") pod \"5261f014-d469-4320-b521-a15cbc87ce02\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " Oct 30 21:51:31.275973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.272452 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls\") pod \"5261f014-d469-4320-b521-a15cbc87ce02\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " Oct 30 21:51:31.275973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.272594 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca\") pod \"5261f014-d469-4320-b521-a15cbc87ce02\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " Oct 30 21:51:31.275973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.272745 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs\") pod \"5261f014-d469-4320-b521-a15cbc87ce02\" (UID: \"5261f014-d469-4320-b521-a15cbc87ce02\") " Oct 30 21:51:31.286909 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.278024 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca" pod "5261f014-d469-4320-b521-a15cbc87ce02" (UID: "5261f014-d469-4320-b521-a15cbc87ce02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:31.289490 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.289048 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls" pod "5261f014-d469-4320-b521-a15cbc87ce02" (UID: "5261f014-d469-4320-b521-a15cbc87ce02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:31.289825 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.289778 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs" pod "5261f014-d469-4320-b521-a15cbc87ce02" (UID: "5261f014-d469-4320-b521-a15cbc87ce02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:31.289934 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.289844 6808 generic.go:356] "Generic (PLEG): container finished" podID="5261f014-d469-4320-b521-a15cbc87ce02" containerID="e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7" exitCode=0 Oct 30 21:51:31.290850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.290287 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" Oct 30 21:51:31.290850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.290668 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" event={"ID":"5261f014-d469-4320-b521-a15cbc87ce02","Type":"ContainerDied","Data":"e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7"} Oct 30 21:51:31.290850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.290720 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp" event={"ID":"5261f014-d469-4320-b521-a15cbc87ce02","Type":"ContainerDied","Data":"6401e9af0396aff5eeb14b237a7618a1c6c0c10152f5d16f088a929626683751"} Oct 30 21:51:31.290850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.290751 6808 scope.go:122] "RemoveContainer" containerID="e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7" Oct 30 21:51:31.293880 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.293821 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7" pod "5261f014-d469-4320-b521-a15cbc87ce02" (UID: "5261f014-d469-4320-b521-a15cbc87ce02"). InnerVolumeSpecName "kube-api-access-6d8c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:31.381349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.381245 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5261f014-d469-4320-b521-a15cbc87ce02-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.381349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.381279 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.381349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.381300 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8c7\" (UniqueName: \"kubernetes.io/projected/5261f014-d469-4320-b521-a15cbc87ce02-kube-api-access-6d8c7\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.381349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.381316 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5261f014-d469-4320-b521-a15cbc87ce02-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.394594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.394221 6808 scope.go:122] "RemoveContainer" containerID="e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7" Oct 30 21:51:31.395001 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:31.394872 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7\": container with ID starting with e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7 not found: ID does not exist" containerID="e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7" Oct 30 21:51:31.395001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.394913 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7"} err="failed to get container status \"e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7\": rpc error: code = NotFound desc = could not find container \"e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7\": container with ID starting with e741f92037a04b56e6caf9455516b8645fe8c5134a727482bf39ee0e209534e7 not found: ID does not exist" Oct 30 21:51:31.544555 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.544528 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:31.631688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.631649 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp"] Oct 30 21:51:31.645491 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.645455 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-db0b54-5bfc7f47c8-dpbmp"] Oct 30 21:51:31.685702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.685671 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca\") pod \"166c9225-d920-4f04-bd04-d043b1228bd0\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " Oct 30 21:51:31.685937 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.685918 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca\") pod \"166c9225-d920-4f04-bd04-d043b1228bd0\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " Oct 30 21:51:31.686065 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.686033 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj\" (UniqueName: \"kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj\") pod \"166c9225-d920-4f04-bd04-d043b1228bd0\" (UID: \"166c9225-d920-4f04-bd04-d043b1228bd0\") " Oct 30 21:51:31.686183 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.686154 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca" pod "166c9225-d920-4f04-bd04-d043b1228bd0" (UID: "166c9225-d920-4f04-bd04-d043b1228bd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:31.686618 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.686571 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.688237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.688201 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca" pod "166c9225-d920-4f04-bd04-d043b1228bd0" (UID: "166c9225-d920-4f04-bd04-d043b1228bd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:31.688901 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.688874 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj" pod "166c9225-d920-4f04-bd04-d043b1228bd0" (UID: "166c9225-d920-4f04-bd04-d043b1228bd0"). InnerVolumeSpecName "kube-api-access-qvllj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:31.787986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.787942 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvllj\" (UniqueName: \"kubernetes.io/projected/166c9225-d920-4f04-bd04-d043b1228bd0-kube-api-access-qvllj\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.787986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.787984 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166c9225-d920-4f04-bd04-d043b1228bd0-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.962027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:31.958268 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5261f014-d469-4320-b521-a15cbc87ce02" path="/var/lib/kubelet/pods/5261f014-d469-4320-b521-a15cbc87ce02/volumes" Oct 30 21:51:32.297573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:32.297497 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" Oct 30 21:51:32.297573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:32.297531 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6" event={"ID":"166c9225-d920-4f04-bd04-d043b1228bd0","Type":"ContainerDied","Data":"ee581eb34443e4f1bc28437ff283ed35023b6b1e42b5529862b87bfa5f3da071"} Oct 30 21:51:32.298048 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:32.297592 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee581eb34443e4f1bc28437ff283ed35023b6b1e42b5529862b87bfa5f3da071" Oct 30 21:51:34.989857 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:34.989753 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" containerID="cri-o://37498662991cfee98111914bc3555167d5f0668465dfa9141d687cede4ecc7ac" gracePeriod=45 Oct 30 21:51:35.017186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.017131 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" containerID="cri-o://e2c7420d2a30c9f9eabaa3b2f1a631687599c9d90aeb67eacec9758412e05ae4" gracePeriod=45 Oct 30 21:51:35.190775 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.190408 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": dial tcp 10.134.0.46:8000: connect: connection refused" Oct 30 21:51:35.222262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.204790 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Oct 30 21:51:35.323719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.323646 6808 generic.go:356] "Generic (PLEG): container finished" podID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerID="e2c7420d2a30c9f9eabaa3b2f1a631687599c9d90aeb67eacec9758412e05ae4" exitCode=0 Oct 30 21:51:35.324026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.323886 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" event={"ID":"8d9f10f5-6db7-4b17-b551-69f7bed48737","Type":"ContainerDied","Data":"e2c7420d2a30c9f9eabaa3b2f1a631687599c9d90aeb67eacec9758412e05ae4"} Oct 30 21:51:35.326021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.325872 6808 generic.go:356] "Generic (PLEG): container finished" podID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerID="37498662991cfee98111914bc3555167d5f0668465dfa9141d687cede4ecc7ac" exitCode=0 Oct 30 21:51:35.326021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.325978 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" event={"ID":"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97","Type":"ContainerDied","Data":"37498662991cfee98111914bc3555167d5f0668465dfa9141d687cede4ecc7ac"} Oct 30 21:51:35.508288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.508259 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:51:35.512475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.512437 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588223 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm\") pod \"8d9f10f5-6db7-4b17-b551-69f7bed48737\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588280 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache\") pod \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588338 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775\" (UniqueName: \"kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775\") pod \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588419 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm\") pod \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588454 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home\") pod \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588543 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs\") pod \"8d9f10f5-6db7-4b17-b551-69f7bed48737\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588589 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir\") pod \"8d9f10f5-6db7-4b17-b551-69f7bed48737\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588633 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir\") pod \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588688 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache\") pod \"8d9f10f5-6db7-4b17-b551-69f7bed48737\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588745 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs\") pod \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\" (UID: \"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588782 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home\") pod \"8d9f10f5-6db7-4b17-b551-69f7bed48737\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " Oct 30 21:51:35.589715 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.588842 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r\" (UniqueName: \"kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r\") pod \"8d9f10f5-6db7-4b17-b551-69f7bed48737\" (UID: \"8d9f10f5-6db7-4b17-b551-69f7bed48737\") " Oct 30 21:51:35.591682 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.591656 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache" pod "8d9f10f5-6db7-4b17-b551-69f7bed48737" (UID: "8d9f10f5-6db7-4b17-b551-69f7bed48737"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.591846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.591666 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir" pod "8d9f10f5-6db7-4b17-b551-69f7bed48737" (UID: "8d9f10f5-6db7-4b17-b551-69f7bed48737"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.591966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.591872 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home" pod "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" (UID: "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.592077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.592057 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache" pod "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" (UID: "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.592222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.592152 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r" pod "8d9f10f5-6db7-4b17-b551-69f7bed48737" (UID: "8d9f10f5-6db7-4b17-b551-69f7bed48737"). InnerVolumeSpecName "kube-api-access-g9h4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.592334 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.592252 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home" pod "8d9f10f5-6db7-4b17-b551-69f7bed48737" (UID: "8d9f10f5-6db7-4b17-b551-69f7bed48737"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.592334 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.592325 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir" pod "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" (UID: "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.601862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.601834 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs" pod "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" (UID: "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:35.601862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.601852 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm" pod "8d9f10f5-6db7-4b17-b551-69f7bed48737" (UID: "8d9f10f5-6db7-4b17-b551-69f7bed48737"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.602425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.601836 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm" pod "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" (UID: "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.602425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.601996 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775" pod "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" (UID: "1c4d339a-ae3c-42c1-85fe-cd80be3e2a97"). InnerVolumeSpecName "kube-api-access-8w775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.604695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.604634 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs" pod "8d9f10f5-6db7-4b17-b551-69f7bed48737" (UID: "8d9f10f5-6db7-4b17-b551-69f7bed48737"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695130 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f10f5-6db7-4b17-b551-69f7bed48737-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695166 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695183 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695196 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695210 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695222 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695236 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9h4r\" (UniqueName: \"kubernetes.io/projected/8d9f10f5-6db7-4b17-b551-69f7bed48737-kube-api-access-g9h4r\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695249 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f10f5-6db7-4b17-b551-69f7bed48737-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695261 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695275 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w775\" (UniqueName: \"kubernetes.io/projected/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-kube-api-access-8w775\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695288 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.695327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:35.695302 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.726698 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:35.726666 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:51:36.302655 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.302559 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" containerID="cri-o://364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3" gracePeriod=45 Oct 30 21:51:36.311677 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.311277 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:51:36.321282 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.321030 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" podUID="d1458c21-ef04-42e2-a973-dff27829548f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:51:36.335234 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.335210 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" Oct 30 21:51:36.335350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.335233 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc" event={"ID":"8d9f10f5-6db7-4b17-b551-69f7bed48737","Type":"ContainerDied","Data":"b6644de59d8cb6eaf6721cb9cd931665c9c6c134abcd138cddf958013995ffc9"} Oct 30 21:51:36.335350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.335287 6808 scope.go:122] "RemoveContainer" containerID="e2c7420d2a30c9f9eabaa3b2f1a631687599c9d90aeb67eacec9758412e05ae4" Oct 30 21:51:36.337430 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.337405 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" event={"ID":"1c4d339a-ae3c-42c1-85fe-cd80be3e2a97","Type":"ContainerDied","Data":"ed14a136efd51d9dfde13eee6801dc95279ee5838cb24f2caae422e79e6adab8"} Oct 30 21:51:36.337502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.337487 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4" Oct 30 21:51:36.426716 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.426678 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc"] Oct 30 21:51:36.431320 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.431264 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/simulator-w1-1a81ae-kserve-7569557765-g2tfc"] Oct 30 21:51:36.436593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.436573 6808 scope.go:122] "RemoveContainer" containerID="37498662991cfee98111914bc3555167d5f0668465dfa9141d687cede4ecc7ac" Oct 30 21:51:36.442425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.442321 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4"] Oct 30 21:51:36.449985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.449865 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-w1-1a81ae-kserve-54f54cfc6c-8dmq4"] Oct 30 21:51:36.652800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.652750 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:51:36.737899 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.737867 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs\") pod \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " Oct 30 21:51:36.738130 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.737911 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm\") pod \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " Oct 30 21:51:36.738130 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.738105 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir\") pod \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " Oct 30 21:51:36.738264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.738175 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home\") pod \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " Oct 30 21:51:36.738264 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.738208 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7\" (UniqueName: \"kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7\") pod \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " Oct 30 21:51:36.738378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.738292 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache\") pod \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\" (UID: \"f885f90c-9fa9-455b-ad81-5225b47fb9f9\") " Oct 30 21:51:36.739275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.739180 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home" pod "f885f90c-9fa9-455b-ad81-5225b47fb9f9" (UID: "f885f90c-9fa9-455b-ad81-5225b47fb9f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.739275 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.739249 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir" pod "f885f90c-9fa9-455b-ad81-5225b47fb9f9" (UID: "f885f90c-9fa9-455b-ad81-5225b47fb9f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.739658 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.739631 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache" pod "f885f90c-9fa9-455b-ad81-5225b47fb9f9" (UID: "f885f90c-9fa9-455b-ad81-5225b47fb9f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.742246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.742221 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs" pod "f885f90c-9fa9-455b-ad81-5225b47fb9f9" (UID: "f885f90c-9fa9-455b-ad81-5225b47fb9f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:36.743647 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.743188 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm" pod "f885f90c-9fa9-455b-ad81-5225b47fb9f9" (UID: "f885f90c-9fa9-455b-ad81-5225b47fb9f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.744213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.744186 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7" pod "f885f90c-9fa9-455b-ad81-5225b47fb9f9" (UID: "f885f90c-9fa9-455b-ad81-5225b47fb9f9"). InnerVolumeSpecName "kube-api-access-jg5n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:36.840432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.840299 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.840432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.840333 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.840432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.840346 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg5n7\" (UniqueName: \"kubernetes.io/projected/f885f90c-9fa9-455b-ad81-5225b47fb9f9-kube-api-access-jg5n7\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.840432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.840361 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.840432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.840374 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f885f90c-9fa9-455b-ad81-5225b47fb9f9-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.840432 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:36.840385 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f885f90c-9fa9-455b-ad81-5225b47fb9f9-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:37.347055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.347019 6808 generic.go:356] "Generic (PLEG): container finished" podID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerID="364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3" exitCode=0 Oct 30 21:51:37.347523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.347070 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" Oct 30 21:51:37.347523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.347095 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" event={"ID":"f885f90c-9fa9-455b-ad81-5225b47fb9f9","Type":"ContainerDied","Data":"364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3"} Oct 30 21:51:37.347523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.347142 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz" event={"ID":"f885f90c-9fa9-455b-ad81-5225b47fb9f9","Type":"ContainerDied","Data":"4a2f08af183ec7f8166e35eec355e8da1361e5748c81b09703a8aeedc0efcbf6"} Oct 30 21:51:37.347523 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.347167 6808 scope.go:122] "RemoveContainer" containerID="364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3" Oct 30 21:51:37.358003 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.357864 6808 scope.go:122] "RemoveContainer" containerID="364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3" Oct 30 21:51:37.358307 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:37.358277 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3\": container with ID starting with 364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3 not found: ID does not exist" containerID="364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3" Oct 30 21:51:37.358374 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.358305 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3"} err="failed to get container status \"364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3\": rpc error: code = NotFound desc = could not find container \"364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3\": container with ID starting with 364768e8605d689efab435743c87df1435dbcb7a1dc02ae62e3e7b2060fda7f3 not found: ID does not exist" Oct 30 21:51:37.373472 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.373448 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz"] Oct 30 21:51:37.415700 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.415668 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/premium-w4-0fa864-kserve-5b7fd4b589-jr4zz"] Oct 30 21:51:37.929588 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.929558 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" path="/var/lib/kubelet/pods/1c4d339a-ae3c-42c1-85fe-cd80be3e2a97/volumes" Oct 30 21:51:37.930013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.929998 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" path="/var/lib/kubelet/pods/8d9f10f5-6db7-4b17-b551-69f7bed48737/volumes" Oct 30 21:51:37.930397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:37.930384 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" path="/var/lib/kubelet/pods/f885f90c-9fa9-455b-ad81-5225b47fb9f9/volumes" Oct 30 21:51:39.975460 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:39.975432 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:51:39.999752 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:39.999723 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume\") pod \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " Oct 30 21:51:40.000185 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.000157 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume" pod "f8987be6-af66-4ccd-892b-6cb2c7c448fc" (UID: "f8987be6-af66-4ccd-892b-6cb2c7c448fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:40.156233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.154682 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh\" (UniqueName: \"kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh\") pod \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\" (UID: \"f8987be6-af66-4ccd-892b-6cb2c7c448fc\") " Oct 30 21:51:40.156233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.155285 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8987be6-af66-4ccd-892b-6cb2c7c448fc-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.162052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.162016 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh" pod "f8987be6-af66-4ccd-892b-6cb2c7c448fc" (UID: "f8987be6-af66-4ccd-892b-6cb2c7c448fc"). InnerVolumeSpecName "kube-api-access-6zvlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.256409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.256329 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvlh\" (UniqueName: \"kubernetes.io/projected/f8987be6-af66-4ccd-892b-6cb2c7c448fc-kube-api-access-6zvlh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.352971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.352902 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9"] Oct 30 21:51:40.358970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.353183 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" containerID="cri-o://8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9" gracePeriod=30 Oct 30 21:51:40.358970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.358374 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv"] Oct 30 21:51:40.358970 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.358596 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" containerID="cri-o://55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239" gracePeriod=30 Oct 30 21:51:40.366855 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.366828 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk"] Oct 30 21:51:40.367029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.367004 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" containerID="cri-o://ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e" gracePeriod=30 Oct 30 21:51:40.391981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.388591 6808 generic.go:356] "Generic (PLEG): container finished" podID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerID="c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a" exitCode=0 Oct 30 21:51:40.391981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.388698 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" event={"ID":"f8987be6-af66-4ccd-892b-6cb2c7c448fc","Type":"ContainerDied","Data":"c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a"} Oct 30 21:51:40.391981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.388758 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" event={"ID":"f8987be6-af66-4ccd-892b-6cb2c7c448fc","Type":"ContainerDied","Data":"7e395325a16efb2e729a1b74286056596a501f2460128e302a96d784c8d8a189"} Oct 30 21:51:40.391981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.388789 6808 scope.go:122] "RemoveContainer" containerID="c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a" Oct 30 21:51:40.391981 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.388923 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2" Oct 30 21:51:40.405036 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.405015 6808 scope.go:122] "RemoveContainer" containerID="c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a" Oct 30 21:51:40.405367 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:40.405335 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a\": container with ID starting with c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a not found: ID does not exist" containerID="c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a" Oct 30 21:51:40.405447 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.405382 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a"} err="failed to get container status \"c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a\": rpc error: code = NotFound desc = could not find container \"c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a\": container with ID starting with c3dcc53d56747f797a19c4a070612f6088df91e32791411dbf4b2770fe2f3f6a not found: ID does not exist" Oct 30 21:51:40.457136 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.457103 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2"] Oct 30 21:51:40.463645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.463616 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-db0b54-747cb5687f-9rph2"] Oct 30 21:51:40.897011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.893392 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc"] Oct 30 21:51:40.897011 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:40.894504 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" podUID="b796528b-ee5b-4b00-8fbd-7f4983eab248" containerName="maas-api" containerID="cri-o://fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7" gracePeriod=30 Oct 30 21:51:41.373669 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.373366 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:51:41.408850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.408124 6808 generic.go:356] "Generic (PLEG): container finished" podID="b796528b-ee5b-4b00-8fbd-7f4983eab248" containerID="fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7" exitCode=0 Oct 30 21:51:41.408850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.408267 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" event={"ID":"b796528b-ee5b-4b00-8fbd-7f4983eab248","Type":"ContainerDied","Data":"fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7"} Oct 30 21:51:41.408850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.408318 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" event={"ID":"b796528b-ee5b-4b00-8fbd-7f4983eab248","Type":"ContainerDied","Data":"f15b8f33b4258b90c94b53310160d72cbc97bc7b50de1aa10d1c23e7878d2dfa"} Oct 30 21:51:41.408850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.408347 6808 scope.go:122] "RemoveContainer" containerID="fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7" Oct 30 21:51:41.408850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.408465 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc" Oct 30 21:51:41.440533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.439511 6808 scope.go:122] "RemoveContainer" containerID="fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7" Oct 30 21:51:41.440959 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:41.440824 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7\": container with ID starting with fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7 not found: ID does not exist" containerID="fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7" Oct 30 21:51:41.440959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.440878 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7"} err="failed to get container status \"fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7\": rpc error: code = NotFound desc = could not find container \"fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7\": container with ID starting with fc41a8864f83546520cec74cb5eb103ff0036d20f6256fdc8614b706822b4ae7 not found: ID does not exist" Oct 30 21:51:41.483923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.483701 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca\") pod \"b796528b-ee5b-4b00-8fbd-7f4983eab248\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " Oct 30 21:51:41.483923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.483811 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls\") pod \"b796528b-ee5b-4b00-8fbd-7f4983eab248\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " Oct 30 21:51:41.483923 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.483878 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs\") pod \"b796528b-ee5b-4b00-8fbd-7f4983eab248\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " Oct 30 21:51:41.484253 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.484012 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9\" (UniqueName: \"kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9\") pod \"b796528b-ee5b-4b00-8fbd-7f4983eab248\" (UID: \"b796528b-ee5b-4b00-8fbd-7f4983eab248\") " Oct 30 21:51:41.484766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.484658 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca" pod "b796528b-ee5b-4b00-8fbd-7f4983eab248" (UID: "b796528b-ee5b-4b00-8fbd-7f4983eab248"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:41.487335 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.487295 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls" pod "b796528b-ee5b-4b00-8fbd-7f4983eab248" (UID: "b796528b-ee5b-4b00-8fbd-7f4983eab248"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:41.487612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.487584 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs" pod "b796528b-ee5b-4b00-8fbd-7f4983eab248" (UID: "b796528b-ee5b-4b00-8fbd-7f4983eab248"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:41.487709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.487628 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9" pod "b796528b-ee5b-4b00-8fbd-7f4983eab248" (UID: "b796528b-ee5b-4b00-8fbd-7f4983eab248"). InnerVolumeSpecName "kube-api-access-tkvg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:41.585425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.585335 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.585425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.585364 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b796528b-ee5b-4b00-8fbd-7f4983eab248-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.585425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.585375 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkvg9\" (UniqueName: \"kubernetes.io/projected/b796528b-ee5b-4b00-8fbd-7f4983eab248-kube-api-access-tkvg9\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.585425 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.585386 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b796528b-ee5b-4b00-8fbd-7f4983eab248-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.734401 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.734370 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc"] Oct 30 21:51:41.745132 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.745103 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1a81ae-669b546df9-zpqpc"] Oct 30 21:51:41.927826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.927794 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b796528b-ee5b-4b00-8fbd-7f4983eab248" path="/var/lib/kubelet/pods/b796528b-ee5b-4b00-8fbd-7f4983eab248/volumes" Oct 30 21:51:41.928299 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:41.928283 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" path="/var/lib/kubelet/pods/f8987be6-af66-4ccd-892b-6cb2c7c448fc/volumes" Oct 30 21:51:42.980128 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:42.980095 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:42.980486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:42.980158 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:44.776597 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:44.776552 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:44.777043 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:44.776619 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:44.886120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:44.876254 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p"] Oct 30 21:51:45.024689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.024649 6808 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:45.024840 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.024729 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:45.741913 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.741879 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:51:45.765271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765235 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765291 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765353 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765385 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765473 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765639 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765596 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765646 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765695 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765678 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.765809 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.765734 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs\") pod \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\" (UID: \"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32\") " Oct 30 21:51:45.768783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.767430 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.768783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.767789 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:45.768783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.768471 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.768783 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.768755 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.769169 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.769134 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.786274 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.786173 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:51:45.788632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.788598 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.792641 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.792604 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.794508 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.794486 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "kube-api-access-kgk4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.800663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.800622 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo" pod "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" (UID: "3f39f5cc-0dcd-49b9-962b-9ea951a2dd32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:45.828688 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.828658 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:51:45.874905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.874861 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875106 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875122 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875131 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875139 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875148 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875157 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875165 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.875375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.875174 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgk4x\" (UniqueName: \"kubernetes.io/projected/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32-kube-api-access-kgk4x\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.975756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.975719 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.975969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.975800 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.975969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.975846 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.975969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.975875 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.975969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.975911 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.975969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.975962 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976042 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976081 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976117 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976145 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976175 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976211 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976220 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.976254 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976243 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976271 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976320 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976373 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976401 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976424 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z\") pod \"72c61b81-7a11-4f8d-852d-0de74e31d752\" (UID: \"72c61b81-7a11-4f8d-852d-0de74e31d752\") " Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976473 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976484 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.976654 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.976513 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert\") pod \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\" (UID: \"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea\") " Oct 30 21:51:45.977098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.977039 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.977098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.977059 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.977098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.977075 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.977098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.977091 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.977314 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.977168 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:45.978376 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.977389 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.978376 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.978011 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.978376 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.978171 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.978633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.978521 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.978633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.978606 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:45.978739 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.978696 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:45.979219 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.979182 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.980579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.980554 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "kube-api-access-766m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.980795 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.980772 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.981421 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.981387 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.981632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.981594 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:45.982242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.982219 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z" pod "72c61b81-7a11-4f8d-852d-0de74e31d752" (UID: "72c61b81-7a11-4f8d-852d-0de74e31d752"). InnerVolumeSpecName "kube-api-access-g4g9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.982388 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:45.982371 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy" pod "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" (UID: "f5ad36b3-d926-44e7-85dd-8d4b0dba50ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:46.078429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078386 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766m2\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-kube-api-access-766m2\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078425 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078440 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078454 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078466 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c61b81-7a11-4f8d-852d-0de74e31d752-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078479 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4g9z\" (UniqueName: \"kubernetes.io/projected/72c61b81-7a11-4f8d-852d-0de74e31d752-kube-api-access-g4g9z\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078494 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078507 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078521 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078534 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078547 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078559 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078572 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c61b81-7a11-4f8d-852d-0de74e31d752-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.078642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.078586 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c61b81-7a11-4f8d-852d-0de74e31d752-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.281222 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.281133 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" podUID="d1458c21-ef04-42e2-a973-dff27829548f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:51:46.439353 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.439320 6808 generic.go:356] "Generic (PLEG): container finished" podID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerID="55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239" exitCode=0 Oct 30 21:51:46.439502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.439374 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" Oct 30 21:51:46.439502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.439399 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" event={"ID":"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32","Type":"ContainerDied","Data":"55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239"} Oct 30 21:51:46.439502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.439456 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv" event={"ID":"3f39f5cc-0dcd-49b9-962b-9ea951a2dd32","Type":"ContainerDied","Data":"0bc3ab4c004a2168c4e364100372ae234c50a3af594af74d294081b3e0a82ac3"} Oct 30 21:51:46.439502 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.439480 6808 scope.go:122] "RemoveContainer" containerID="55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239" Oct 30 21:51:46.440879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.440854 6808 generic.go:356] "Generic (PLEG): container finished" podID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerID="8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9" exitCode=0 Oct 30 21:51:46.440982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.440916 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" Oct 30 21:51:46.440982 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.440944 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" event={"ID":"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea","Type":"ContainerDied","Data":"8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9"} Oct 30 21:51:46.441131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.441020 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9" event={"ID":"f5ad36b3-d926-44e7-85dd-8d4b0dba50ea","Type":"ContainerDied","Data":"8aee7a7b29984a3f16384a86c9db5ea7f83be09e23f77fd84bd77c3ef0e41e05"} Oct 30 21:51:46.442831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.442805 6808 generic.go:356] "Generic (PLEG): container finished" podID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerID="ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e" exitCode=0 Oct 30 21:51:46.442933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.442878 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" event={"ID":"72c61b81-7a11-4f8d-852d-0de74e31d752","Type":"ContainerDied","Data":"ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e"} Oct 30 21:51:46.442933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.442912 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" event={"ID":"72c61b81-7a11-4f8d-852d-0de74e31d752","Type":"ContainerDied","Data":"9a164e25137797877fa9ca83ca2bdac66c389798873eb251706db56438fe23e9"} Oct 30 21:51:46.443072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.443012 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk" Oct 30 21:51:46.450689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.450671 6808 scope.go:122] "RemoveContainer" containerID="3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.460903 6808 scope.go:122] "RemoveContainer" containerID="55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:46.461222 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239\": container with ID starting with 55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239 not found: ID does not exist" containerID="55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.461255 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239"} err="failed to get container status \"55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239\": rpc error: code = NotFound desc = could not find container \"55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239\": container with ID starting with 55851ca268ebb156a645a5cf5f1257db84def65bda51fd26a7dfc052e5183239 not found: ID does not exist" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.461278 6808 scope.go:122] "RemoveContainer" containerID="3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:46.461531 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3\": container with ID starting with 3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3 not found: ID does not exist" containerID="3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.461558 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3"} err="failed to get container status \"3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3\": rpc error: code = NotFound desc = could not find container \"3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3\": container with ID starting with 3e2ea0787d759b0b232df9da080b8f38292cd3ac87f794ef019ff247a9dd96e3 not found: ID does not exist" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.461578 6808 scope.go:122] "RemoveContainer" containerID="8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9" Oct 30 21:51:46.464661 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.463731 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv"] Oct 30 21:51:46.470153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.470133 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-zdtkv"] Oct 30 21:51:46.473907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.473884 6808 scope.go:122] "RemoveContainer" containerID="8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9" Oct 30 21:51:46.474228 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:46.474205 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9\": container with ID starting with 8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9 not found: ID does not exist" containerID="8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9" Oct 30 21:51:46.474324 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.474232 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9"} err="failed to get container status \"8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9\": rpc error: code = NotFound desc = could not find container \"8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9\": container with ID starting with 8ba35024ad776725bd87b0a795ff0d10b94bc8aa35814284810b626c29c478d9 not found: ID does not exist" Oct 30 21:51:46.474324 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.474246 6808 scope.go:122] "RemoveContainer" containerID="ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e" Oct 30 21:51:46.484995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.484766 6808 scope.go:122] "RemoveContainer" containerID="ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e" Oct 30 21:51:46.485068 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:46.485053 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e\": container with ID starting with ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e not found: ID does not exist" containerID="ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e" Oct 30 21:51:46.485129 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.485076 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e"} err="failed to get container status \"ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e\": rpc error: code = NotFound desc = could not find container \"ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e\": container with ID starting with ca5c9d58ef4d0a4fc6592cf25641a612089036b2a792ab232a2bb4b9ae268c1e not found: ID does not exist" Oct 30 21:51:46.489714 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.489690 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk"] Oct 30 21:51:46.493204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.493181 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-7hgwk"] Oct 30 21:51:46.504888 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.504869 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9"] Oct 30 21:51:46.509719 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:46.509702 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-6rvv9"] Oct 30 21:51:47.930601 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:47.930554 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" path="/var/lib/kubelet/pods/3f39f5cc-0dcd-49b9-962b-9ea951a2dd32/volumes" Oct 30 21:51:47.931255 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:47.931238 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" path="/var/lib/kubelet/pods/72c61b81-7a11-4f8d-852d-0de74e31d752/volumes" Oct 30 21:51:47.931805 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:47.931775 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" path="/var/lib/kubelet/pods/f5ad36b3-d926-44e7-85dd-8d4b0dba50ea/volumes" Oct 30 21:51:48.305039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:48.303749 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7"] Oct 30 21:51:48.305039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:48.304987 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" containerID="cri-o://bae25150204a2ea3e5f122cdbe596da7b30af0618087b7c6d276c8d672835817" gracePeriod=30 Oct 30 21:51:48.331478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:48.309033 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w"] Oct 30 21:51:48.331478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:48.309266 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" containerID="cri-o://67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0" gracePeriod=30 Oct 30 21:51:49.804093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:49.804049 6808 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:49.804554 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:49.804118 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:50.563678 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.563649 6808 generic.go:356] "Generic (PLEG): container finished" podID="d1458c21-ef04-42e2-a973-dff27829548f" containerID="9dd3947b1c865d2701183ef84ffcc564521fb869468d0ccf552db9a83a68fbd1" exitCode=0 Oct 30 21:51:50.563823 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.563723 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" event={"ID":"d1458c21-ef04-42e2-a973-dff27829548f","Type":"ContainerDied","Data":"9dd3947b1c865d2701183ef84ffcc564521fb869468d0ccf552db9a83a68fbd1"} Oct 30 21:51:50.588198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.588173 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:51:50.775882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.775848 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm\" (UniqueName: \"kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm\") pod \"d1458c21-ef04-42e2-a973-dff27829548f\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " Oct 30 21:51:50.776068 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.775900 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume\") pod \"d1458c21-ef04-42e2-a973-dff27829548f\" (UID: \"d1458c21-ef04-42e2-a973-dff27829548f\") " Oct 30 21:51:50.776324 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.776298 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume" pod "d1458c21-ef04-42e2-a973-dff27829548f" (UID: "d1458c21-ef04-42e2-a973-dff27829548f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:50.778074 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.778052 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm" pod "d1458c21-ef04-42e2-a973-dff27829548f" (UID: "d1458c21-ef04-42e2-a973-dff27829548f"). InnerVolumeSpecName "kube-api-access-x28tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:50.877522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.877428 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28tm\" (UniqueName: \"kubernetes.io/projected/d1458c21-ef04-42e2-a973-dff27829548f-kube-api-access-x28tm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.877522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:50.877468 6808 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1458c21-ef04-42e2-a973-dff27829548f-plugins-config-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:51.571064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.571022 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" event={"ID":"d1458c21-ef04-42e2-a973-dff27829548f","Type":"ContainerDied","Data":"b0726fcc0dcb6ab081c217723c901a943ccb11eebcb0df07074db3bc0dc08eeb"} Oct 30 21:51:51.571064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.571049 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv" Oct 30 21:51:51.571256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.571109 6808 scope.go:122] "RemoveContainer" containerID="9dd3947b1c865d2701183ef84ffcc564521fb869468d0ccf552db9a83a68fbd1" Oct 30 21:51:51.606262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.606205 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv"] Oct 30 21:51:51.614246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.614212 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1a81ae-84df9f495-9p9hv"] Oct 30 21:51:51.886249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.856207 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v"] Oct 30 21:51:51.886249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.877105 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" containerID="cri-o://adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3" gracePeriod=30 Oct 30 21:51:51.886249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.880837 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg"] Oct 30 21:51:51.886249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.881062 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" containerID="cri-o://66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926" gracePeriod=30 Oct 30 21:51:51.886249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.884525 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s"] Oct 30 21:51:51.886249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.884743 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" containerID="cri-o://5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb" gracePeriod=30 Oct 30 21:51:51.932595 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:51.932537 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1458c21-ef04-42e2-a973-dff27829548f" path="/var/lib/kubelet/pods/d1458c21-ef04-42e2-a973-dff27829548f/volumes" Oct 30 21:51:52.330063 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.329979 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56"] Oct 30 21:51:52.334556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.330845 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" Oct 30 21:51:52.335194 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335167 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" Oct 30 21:51:52.335366 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335353 6808 cpu_manager.go:414] "containerMap: removing container" podUID="d1458c21-ef04-42e2-a973-dff27829548f" containerName="payload-pre-processing" Oct 30 21:51:52.335463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335451 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1458c21-ef04-42e2-a973-dff27829548f" containerName="payload-pre-processing" Oct 30 21:51:52.335541 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335532 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" Oct 30 21:51:52.335613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335603 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" Oct 30 21:51:52.335698 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335689 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerName="payload-pre-processing" Oct 30 21:51:52.335796 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335784 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerName="payload-pre-processing" Oct 30 21:51:52.335872 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335863 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" Oct 30 21:51:52.335963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.335935 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" Oct 30 21:51:52.336071 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336061 6808 cpu_manager.go:414] "containerMap: removing container" podUID="166c9225-d920-4f04-bd04-d043b1228bd0" containerName="revoke-keys" Oct 30 21:51:52.336138 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336129 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166c9225-d920-4f04-bd04-d043b1228bd0" containerName="revoke-keys" Oct 30 21:51:52.336220 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336211 6808 cpu_manager.go:414] "containerMap: removing container" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" Oct 30 21:51:52.336285 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336277 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" Oct 30 21:51:52.336382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336371 6808 cpu_manager.go:414] "containerMap: removing container" podUID="5261f014-d469-4320-b521-a15cbc87ce02" containerName="maas-api" Oct 30 21:51:52.336474 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336463 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5261f014-d469-4320-b521-a15cbc87ce02" containerName="maas-api" Oct 30 21:51:52.336567 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336557 6808 cpu_manager.go:414] "containerMap: removing container" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" Oct 30 21:51:52.336662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336651 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" Oct 30 21:51:52.336751 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336741 6808 cpu_manager.go:414] "containerMap: removing container" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" Oct 30 21:51:52.336839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336828 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" Oct 30 21:51:52.336975 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.336944 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b796528b-ee5b-4b00-8fbd-7f4983eab248" containerName="maas-api" Oct 30 21:51:52.337080 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337068 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b796528b-ee5b-4b00-8fbd-7f4983eab248" containerName="maas-api" Oct 30 21:51:52.337356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337342 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="166c9225-d920-4f04-bd04-d043b1228bd0" containerName="revoke-keys" Oct 30 21:51:52.337475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337463 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c61b81-7a11-4f8d-852d-0de74e31d752" containerName="istio-proxy" Oct 30 21:51:52.337575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337565 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="5261f014-d469-4320-b521-a15cbc87ce02" containerName="maas-api" Oct 30 21:51:52.337677 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337666 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9f10f5-6db7-4b17-b551-69f7bed48737" containerName="main" Oct 30 21:51:52.337773 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337762 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4d339a-ae3c-42c1-85fe-cd80be3e2a97" containerName="main" Oct 30 21:51:52.337884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337874 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" Oct 30 21:51:52.338001 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.337990 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b796528b-ee5b-4b00-8fbd-7f4983eab248" containerName="maas-api" Oct 30 21:51:52.338127 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.338102 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1458c21-ef04-42e2-a973-dff27829548f" containerName="payload-pre-processing" Oct 30 21:51:52.338226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.338215 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ad36b3-d926-44e7-85dd-8d4b0dba50ea" containerName="istio-proxy" Oct 30 21:51:52.338305 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.338296 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8987be6-af66-4ccd-892b-6cb2c7c448fc" containerName="payload-pre-processing" Oct 30 21:51:52.338375 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.338367 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f885f90c-9fa9-455b-ad81-5225b47fb9f9" containerName="main" Oct 30 21:51:52.347211 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.347185 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.348193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.348164 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56"] Oct 30 21:51:52.411983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.411082 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.411983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.411241 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.411983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.411308 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ztjx\" (UniqueName: \"kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.439974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.439268 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:52.439974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.439340 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:52.519974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.514651 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.519974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.514745 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztjx\" (UniqueName: \"kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.519974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.514833 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.519974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.515685 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.554969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.554569 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztjx\" (UniqueName: \"kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.554969 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.554882 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e6911f-hch56\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:52.727526 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:52.726113 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:53.018891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.018865 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56"] Oct 30 21:51:53.021402 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:51:53.021365 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd84627_40a3_472e_8d9c_b1aa22437cd4.slice/crio-a71b072d3d3927550b8b79505bc02c2669207d316e293469b7331abe87bf1f10 WatchSource:0}: Error finding container a71b072d3d3927550b8b79505bc02c2669207d316e293469b7331abe87bf1f10: Status 404 returned error can't find the container with id a71b072d3d3927550b8b79505bc02c2669207d316e293469b7331abe87bf1f10 Oct 30 21:51:53.634090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.634050 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" event={"ID":"7dd84627-40a3-472e-8d9c-b1aa22437cd4","Type":"ContainerStarted","Data":"1bbe6ab926b6b6b666a5837aad77c082d56983e197807146b45ef561477be6e4"} Oct 30 21:51:53.634258 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.634120 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" event={"ID":"7dd84627-40a3-472e-8d9c-b1aa22437cd4","Type":"ContainerStarted","Data":"a71b072d3d3927550b8b79505bc02c2669207d316e293469b7331abe87bf1f10"} Oct 30 21:51:53.634901 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.634834 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:51:53.635713 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.635689 6808 generic.go:356] "Generic (PLEG): container finished" podID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerID="67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0" exitCode=0 Oct 30 21:51:53.635839 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.635817 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" event={"ID":"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8","Type":"ContainerDied","Data":"67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0"} Oct 30 21:51:53.635942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.635860 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" event={"ID":"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8","Type":"ContainerDied","Data":"1e351c9674b4ad461c9bbdd1bb9fcdea17dc5a50bf6e9d41f423de7804e92634"} Oct 30 21:51:53.635942 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.635889 6808 scope.go:122] "RemoveContainer" containerID="67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0" Oct 30 21:51:53.638166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.638135 6808 generic.go:356] "Generic (PLEG): container finished" podID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerID="bae25150204a2ea3e5f122cdbe596da7b30af0618087b7c6d276c8d672835817" exitCode=0 Oct 30 21:51:53.638239 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.638164 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" event={"ID":"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1","Type":"ContainerDied","Data":"bae25150204a2ea3e5f122cdbe596da7b30af0618087b7c6d276c8d672835817"} Oct 30 21:51:53.646676 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.646499 6808 scope.go:122] "RemoveContainer" containerID="67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0" Oct 30 21:51:53.646906 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:53.646830 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0\": container with ID starting with 67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0 not found: ID does not exist" containerID="67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0" Oct 30 21:51:53.646906 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.646865 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0"} err="failed to get container status \"67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0\": rpc error: code = NotFound desc = could not find container \"67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0\": container with ID starting with 67009deded9f279211d2c5beeac9e7be82208a73641e9a6aa30d357d04223be0 not found: ID does not exist" Oct 30 21:51:53.656177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.656130 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" podStartSLOduration=1.656113238 podStartE2EDuration="1.656113238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:53.652099619 +0000 UTC m=+1375.847790751" watchObservedRunningTime="2026-10-30 21:51:53.656113238 +0000 UTC m=+1375.851804360" Oct 30 21:51:53.687141 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.687111 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:51:53.734499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734469 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.734651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734607 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.734651 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734633 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.734791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734670 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.734791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734697 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.734907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734816 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.734907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734872 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.734907 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734901 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734934 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734981 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734989 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.734987 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735009 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735045 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735082 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.735121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735112 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735166 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735174 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735187 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735214 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735259 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735299 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert\") pod \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\" (UID: \"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1\") " Oct 30 21:51:53.735524 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735340 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket\") pod \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\" (UID: \"b2439ab7-e15d-4bbf-b6b1-affdb57e26f8\") " Oct 30 21:51:53.735893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735692 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.735893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735710 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.735893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735719 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.735893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735728 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.735893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.735737 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.736173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.736156 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:53.736515 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.736488 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:53.736731 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.736708 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.737837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.737625 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.737962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.737855 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.738201 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.738177 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.738581 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.738561 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.738683 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.738665 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "kube-api-access-6zm9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.739623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.739500 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "kube-api-access-kcj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.739623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.739514 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:53.739623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.739590 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy" pod "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" (UID: "b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.740092 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.740057 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:53.741832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.741801 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token" pod "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" (UID: "b2439ab7-e15d-4bbf-b6b1-affdb57e26f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.837143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837109 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837139 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcj9b\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-kube-api-access-kcj9b\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837150 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837159 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837168 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837176 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837185 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837193 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837202 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837210 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837219 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zm9j\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-kube-api-access-6zm9j\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837227 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.837367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:53.837235 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:54.645963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.645927 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w" Oct 30 21:51:54.647735 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.647711 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" Oct 30 21:51:54.647883 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.647838 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7" event={"ID":"b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1","Type":"ContainerDied","Data":"91744b467c4ddd9d7f9fb765915c0355ecb629fbc44308ffc5eb8abdc9bf87ba"} Oct 30 21:51:54.647980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.647962 6808 scope.go:122] "RemoveContainer" containerID="bae25150204a2ea3e5f122cdbe596da7b30af0618087b7c6d276c8d672835817" Oct 30 21:51:54.650318 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.650289 6808 generic.go:356] "Generic (PLEG): container finished" podID="7dd84627-40a3-472e-8d9c-b1aa22437cd4" containerID="1bbe6ab926b6b6b666a5837aad77c082d56983e197807146b45ef561477be6e4" exitCode=0 Oct 30 21:51:54.650475 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.650332 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" event={"ID":"7dd84627-40a3-472e-8d9c-b1aa22437cd4","Type":"ContainerDied","Data":"1bbe6ab926b6b6b666a5837aad77c082d56983e197807146b45ef561477be6e4"} Oct 30 21:51:54.685625 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.685596 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7"] Oct 30 21:51:54.695351 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.695326 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-8bcc7"] Oct 30 21:51:54.708603 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.708570 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w"] Oct 30 21:51:54.715938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:54.715915 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-r5q2w"] Oct 30 21:51:55.798311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.798289 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:55.854670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.854643 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca\") pod \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " Oct 30 21:51:55.854853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.854771 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca\") pod \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " Oct 30 21:51:55.854917 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.854875 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx\" (UniqueName: \"kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx\") pod \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\" (UID: \"7dd84627-40a3-472e-8d9c-b1aa22437cd4\") " Oct 30 21:51:55.855122 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.855100 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca" pod "7dd84627-40a3-472e-8d9c-b1aa22437cd4" (UID: "7dd84627-40a3-472e-8d9c-b1aa22437cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:55.855299 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.855279 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:55.856893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.856868 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca" pod "7dd84627-40a3-472e-8d9c-b1aa22437cd4" (UID: "7dd84627-40a3-472e-8d9c-b1aa22437cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:55.857064 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.857021 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx" pod "7dd84627-40a3-472e-8d9c-b1aa22437cd4" (UID: "7dd84627-40a3-472e-8d9c-b1aa22437cd4"). InnerVolumeSpecName "kube-api-access-7ztjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:55.926905 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.926849 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" path="/var/lib/kubelet/pods/b2439ab7-e15d-4bbf-b6b1-affdb57e26f8/volumes" Oct 30 21:51:55.927336 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.927322 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" path="/var/lib/kubelet/pods/b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1/volumes" Oct 30 21:51:55.956021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.955996 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd84627-40a3-472e-8d9c-b1aa22437cd4-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:55.956021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:55.956019 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ztjx\" (UniqueName: \"kubernetes.io/projected/7dd84627-40a3-472e-8d9c-b1aa22437cd4-kube-api-access-7ztjx\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:56.667228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:56.667180 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" event={"ID":"7dd84627-40a3-472e-8d9c-b1aa22437cd4","Type":"ContainerDied","Data":"a71b072d3d3927550b8b79505bc02c2669207d316e293469b7331abe87bf1f10"} Oct 30 21:51:56.667228 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:56.667209 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56" Oct 30 21:51:56.667455 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:56.667242 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71b072d3d3927550b8b79505bc02c2669207d316e293469b7331abe87bf1f10" Oct 30 21:51:57.209296 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.209267 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:51:57.273914 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.273886 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.273940 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274006 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274039 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274093 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274132 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274229 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274320 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.274386 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274351 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token\") pod \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\" (UID: \"7d6e6998-bee0-47aa-9fe6-87168a547b1e\") " Oct 30 21:51:57.276291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.274905 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.276291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.275037 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.276291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.275214 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.276291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.275303 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:57.276291 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.276257 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.279972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.278018 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.279972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.279923 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.280156 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.280004 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:57.281499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.281478 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27" pod "7d6e6998-bee0-47aa-9fe6-87168a547b1e" (UID: "7d6e6998-bee0-47aa-9fe6-87168a547b1e"). InnerVolumeSpecName "kube-api-access-dmt27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.289979 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.286444 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:51:57.290397 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.290373 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.397732 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.397807 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.397871 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.397909 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.397942 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398011 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398153 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398212 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398271 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398356 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398671 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398744 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398785 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.398854 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.399151 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.399194 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.399235 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data\") pod \"edb0c9aa-72a0-40ab-a90f-869389caa158\" (UID: \"edb0c9aa-72a0-40ab-a90f-869389caa158\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.399465 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs\") pod \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\" (UID: \"ebc26646-e9c2-4096-90ad-4b0df42ba0e1\") " Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.403542 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.404166 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.404507 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.411525 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.404981 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:57.416186 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.416156 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.416379 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.416346 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.417167 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.417184 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.418914 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.418964 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.418981 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419003 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419019 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmt27\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-kube-api-access-dmt27\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419042 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419060 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419079 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419094 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6e6998-bee0-47aa-9fe6-87168a547b1e-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.419996 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.421644 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.421736 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:57.426246 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.426057 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "kube-api-access-xnj58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.431576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.431398 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.431576 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.431543 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "kube-api-access-j9wqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.433507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.433468 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:57.433744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.433722 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.433832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.433743 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy" pod "edb0c9aa-72a0-40ab-a90f-869389caa158" (UID: "edb0c9aa-72a0-40ab-a90f-869389caa158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.433832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.433745 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token" pod "ebc26646-e9c2-4096-90ad-4b0df42ba0e1" (UID: "ebc26646-e9c2-4096-90ad-4b0df42ba0e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.520507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520416 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520451 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520465 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520478 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520491 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520507 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520504 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520516 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520529 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520540 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520552 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wqf\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-kube-api-access-j9wqf\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520566 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520578 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520590 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb0c9aa-72a0-40ab-a90f-869389caa158-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520601 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520616 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520629 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-kube-api-access-xnj58\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520641 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb0c9aa-72a0-40ab-a90f-869389caa158-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.520831 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.520652 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc26646-e9c2-4096-90ad-4b0df42ba0e1-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.675464 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.675419 6808 generic.go:356] "Generic (PLEG): container finished" podID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerID="66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926" exitCode=0 Oct 30 21:51:57.675632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.675477 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" Oct 30 21:51:57.675632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.675491 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" event={"ID":"7d6e6998-bee0-47aa-9fe6-87168a547b1e","Type":"ContainerDied","Data":"66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926"} Oct 30 21:51:57.675632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.675568 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg" event={"ID":"7d6e6998-bee0-47aa-9fe6-87168a547b1e","Type":"ContainerDied","Data":"cd03ae3076d8c1fda167fca2fd276455a9ca6201c3145104f04998174c156f7e"} Oct 30 21:51:57.675632 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.675589 6808 scope.go:122] "RemoveContainer" containerID="66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926" Oct 30 21:51:57.677198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.677158 6808 generic.go:356] "Generic (PLEG): container finished" podID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerID="adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3" exitCode=0 Oct 30 21:51:57.677317 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.677225 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" Oct 30 21:51:57.677317 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.677261 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" event={"ID":"edb0c9aa-72a0-40ab-a90f-869389caa158","Type":"ContainerDied","Data":"adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3"} Oct 30 21:51:57.677317 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.677299 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v" event={"ID":"edb0c9aa-72a0-40ab-a90f-869389caa158","Type":"ContainerDied","Data":"32c35c91ad001e9eb3448e019b02977a28d26d876d891d1c81a38dab4b1421ae"} Oct 30 21:51:57.682002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.678990 6808 generic.go:356] "Generic (PLEG): container finished" podID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerID="5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb" exitCode=0 Oct 30 21:51:57.682002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.679067 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" event={"ID":"ebc26646-e9c2-4096-90ad-4b0df42ba0e1","Type":"ContainerDied","Data":"5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb"} Oct 30 21:51:57.682002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.679110 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" event={"ID":"ebc26646-e9c2-4096-90ad-4b0df42ba0e1","Type":"ContainerDied","Data":"1adddbebec7387f8556598f6b4aef3074d288fc1c50915b4e1013b38d6e6dc10"} Oct 30 21:51:57.682002 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.679196 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s" Oct 30 21:51:57.687544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.687517 6808 scope.go:122] "RemoveContainer" containerID="66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926" Oct 30 21:51:57.687839 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:57.687815 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926\": container with ID starting with 66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926 not found: ID does not exist" containerID="66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926" Oct 30 21:51:57.687932 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.687852 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926"} err="failed to get container status \"66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926\": rpc error: code = NotFound desc = could not find container \"66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926\": container with ID starting with 66c66c683982fa523590b5372441294236c93928a2382c435c6640887a8c6926 not found: ID does not exist" Oct 30 21:51:57.687932 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.687874 6808 scope.go:122] "RemoveContainer" containerID="adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3" Oct 30 21:51:57.700838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.697380 6808 scope.go:122] "RemoveContainer" containerID="adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3" Oct 30 21:51:57.700838 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:57.697974 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3\": container with ID starting with adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3 not found: ID does not exist" containerID="adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3" Oct 30 21:51:57.700838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.698001 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3"} err="failed to get container status \"adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3\": rpc error: code = NotFound desc = could not find container \"adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3\": container with ID starting with adb09082e02585a59f23e461af0d6a818132dffaac71c330f9fc8b46904c95a3 not found: ID does not exist" Oct 30 21:51:57.700838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.698022 6808 scope.go:122] "RemoveContainer" containerID="5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb" Oct 30 21:51:57.707493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.707477 6808 scope.go:122] "RemoveContainer" containerID="5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb" Oct 30 21:51:57.707781 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:51:57.707763 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb\": container with ID starting with 5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb not found: ID does not exist" containerID="5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb" Oct 30 21:51:57.707828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.707788 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb"} err="failed to get container status \"5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb\": rpc error: code = NotFound desc = could not find container \"5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb\": container with ID starting with 5cc905d2700812880965458810401ecd0ac0266ecca34f1e5cb6e74e8a3264fb not found: ID does not exist" Oct 30 21:51:57.711438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.711417 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg"] Oct 30 21:51:57.717546 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.717524 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-2wwrg"] Oct 30 21:51:57.731399 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.731379 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s"] Oct 30 21:51:57.741611 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.741594 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-wf64s"] Oct 30 21:51:57.755915 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.755889 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v"] Oct 30 21:51:57.760684 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.760656 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-8dh7v"] Oct 30 21:51:57.964976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.957067 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" path="/var/lib/kubelet/pods/7d6e6998-bee0-47aa-9fe6-87168a547b1e/volumes" Oct 30 21:51:57.964976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.957721 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" path="/var/lib/kubelet/pods/ebc26646-e9c2-4096-90ad-4b0df42ba0e1/volumes" Oct 30 21:51:57.964976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:57.958664 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" path="/var/lib/kubelet/pods/edb0c9aa-72a0-40ab-a90f-869389caa158/volumes" Oct 30 21:51:59.593450 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:59.593415 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9"] Oct 30 21:51:59.593933 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:59.593621 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" containerID="cri-o://dd3e450d0fe939faa6663a1e51bfdf745b6e2b45e9fbb63c78f8aa19d7ab5787" gracePeriod=30 Oct 30 21:51:59.963550 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:51:59.963505 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" containerID="cri-o://f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b" gracePeriod=45 Oct 30 21:52:00.215390 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.215326 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:52:00.274522 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274489 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir\") pod \"f2ffe4f8-d41c-4236-a34e-19a756febd79\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " Oct 30 21:52:00.274711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274589 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache\") pod \"f2ffe4f8-d41c-4236-a34e-19a756febd79\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " Oct 30 21:52:00.274711 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274651 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs\") pod \"f2ffe4f8-d41c-4236-a34e-19a756febd79\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " Oct 30 21:52:00.274838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274746 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm\") pod \"f2ffe4f8-d41c-4236-a34e-19a756febd79\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " Oct 30 21:52:00.274838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274795 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz\" (UniqueName: \"kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz\") pod \"f2ffe4f8-d41c-4236-a34e-19a756febd79\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " Oct 30 21:52:00.274838 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274798 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir" pod "f2ffe4f8-d41c-4236-a34e-19a756febd79" (UID: "f2ffe4f8-d41c-4236-a34e-19a756febd79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.275057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.274848 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home\") pod \"f2ffe4f8-d41c-4236-a34e-19a756febd79\" (UID: \"f2ffe4f8-d41c-4236-a34e-19a756febd79\") " Oct 30 21:52:00.275193 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.275172 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home" pod "f2ffe4f8-d41c-4236-a34e-19a756febd79" (UID: "f2ffe4f8-d41c-4236-a34e-19a756febd79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.275258 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.275218 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.275448 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.275422 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache" pod "f2ffe4f8-d41c-4236-a34e-19a756febd79" (UID: "f2ffe4f8-d41c-4236-a34e-19a756febd79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.284327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.284110 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs" pod "f2ffe4f8-d41c-4236-a34e-19a756febd79" (UID: "f2ffe4f8-d41c-4236-a34e-19a756febd79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:00.284327 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.284132 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm" pod "f2ffe4f8-d41c-4236-a34e-19a756febd79" (UID: "f2ffe4f8-d41c-4236-a34e-19a756febd79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.289115 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.289078 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz" pod "f2ffe4f8-d41c-4236-a34e-19a756febd79" (UID: "f2ffe4f8-d41c-4236-a34e-19a756febd79"). InnerVolumeSpecName "kube-api-access-2kxmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:00.376549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.376522 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxmz\" (UniqueName: \"kubernetes.io/projected/f2ffe4f8-d41c-4236-a34e-19a756febd79-kube-api-access-2kxmz\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.376549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.376548 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.376549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.376558 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.376768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.376567 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ffe4f8-d41c-4236-a34e-19a756febd79-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.376768 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.376575 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ffe4f8-d41c-4236-a34e-19a756febd79-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.740898 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.740861 6808 generic.go:356] "Generic (PLEG): container finished" podID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerID="f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b" exitCode=0 Oct 30 21:52:00.741378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.740921 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" Oct 30 21:52:00.741378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.740977 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" event={"ID":"f2ffe4f8-d41c-4236-a34e-19a756febd79","Type":"ContainerDied","Data":"f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b"} Oct 30 21:52:00.741378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.741033 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p" event={"ID":"f2ffe4f8-d41c-4236-a34e-19a756febd79","Type":"ContainerDied","Data":"0abe9f7ceac11047245b2c5c23c3e313c7fd8ed9e3e113202e31ee98c3bd1410"} Oct 30 21:52:00.741378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.741060 6808 scope.go:122] "RemoveContainer" containerID="f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b" Oct 30 21:52:00.759443 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.759411 6808 scope.go:122] "RemoveContainer" containerID="f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b" Oct 30 21:52:00.759778 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:52:00.759757 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b\": container with ID starting with f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b not found: ID does not exist" containerID="f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b" Oct 30 21:52:00.759856 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.759791 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b"} err="failed to get container status \"f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b\": rpc error: code = NotFound desc = could not find container \"f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b\": container with ID starting with f5b181f36cdc7e813030e5c9d4cd2ea35f82f3c6c19bb246e869ffb4ae364e6b not found: ID does not exist" Oct 30 21:52:00.770489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.770463 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p"] Oct 30 21:52:00.776442 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:00.776422 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/distinct-2-w5-caa3dc-kserve-77f74c9c46-jlg2p"] Oct 30 21:52:01.927965 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:01.927918 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" path="/var/lib/kubelet/pods/f2ffe4f8-d41c-4236-a34e-19a756febd79/volumes" Oct 30 21:52:04.721657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.721630 6808 scope.go:122] "RemoveContainer" containerID="a05f08f8ff52cef1fb53c70356968f8f09272497fa00a352fafc4fd8a03f321c" Oct 30 21:52:04.779026 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.778995 6808 generic.go:356] "Generic (PLEG): container finished" podID="d2f3f5a4-1224-4297-91da-291efc553540" containerID="dd3e450d0fe939faa6663a1e51bfdf745b6e2b45e9fbb63c78f8aa19d7ab5787" exitCode=0 Oct 30 21:52:04.779155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.779136 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" event={"ID":"d2f3f5a4-1224-4297-91da-291efc553540","Type":"ContainerDied","Data":"dd3e450d0fe939faa6663a1e51bfdf745b6e2b45e9fbb63c78f8aa19d7ab5787"} Oct 30 21:52:04.865378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.865316 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:52:04.927756 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.927725 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.927929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.927766 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.927929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.927802 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.927929 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.927871 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.928154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.927988 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.928154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928019 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.928154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928054 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.928154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928066 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.928154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928091 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.928154 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928126 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data\") pod \"d2f3f5a4-1224-4297-91da-291efc553540\" (UID: \"d2f3f5a4-1224-4297-91da-291efc553540\") " Oct 30 21:52:04.928482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928258 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.928482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928440 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.928594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928558 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.928594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928577 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.928594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928592 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.929083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928719 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:04.929083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.928791 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.930153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.930123 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.930575 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.930544 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "kube-api-access-dl89x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.930709 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.930692 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:04.930928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:04.930911 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token" pod "d2f3f5a4-1224-4297-91da-291efc553540" (UID: "d2f3f5a4-1224-4297-91da-291efc553540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:05.029653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.029620 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3f5a4-1224-4297-91da-291efc553540-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.029653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.029651 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3f5a4-1224-4297-91da-291efc553540-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.029653 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.029660 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.029879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.029670 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl89x\" (UniqueName: \"kubernetes.io/projected/d2f3f5a4-1224-4297-91da-291efc553540-kube-api-access-dl89x\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.029879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.029680 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.029879 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.029689 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3f5a4-1224-4297-91da-291efc553540-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.700087 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:52:05.700051 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:52:05.786681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.786648 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" event={"ID":"d2f3f5a4-1224-4297-91da-291efc553540","Type":"ContainerDied","Data":"a7f4381f4ea03cfd0a57a6128d76fbde388906f4dce31f7df506fd55f37a3755"} Oct 30 21:52:05.787177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.786684 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9" Oct 30 21:52:05.787177 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.786711 6808 scope.go:122] "RemoveContainer" containerID="dd3e450d0fe939faa6663a1e51bfdf745b6e2b45e9fbb63c78f8aa19d7ab5787" Oct 30 21:52:05.816533 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.816505 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9"] Oct 30 21:52:05.827478 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.827452 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-7kbv9"] Oct 30 21:52:05.927686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:05.927659 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f3f5a4-1224-4297-91da-291efc553540" path="/var/lib/kubelet/pods/d2f3f5a4-1224-4297-91da-291efc553540/volumes" Oct 30 21:52:07.424727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:07.424685 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw"] Oct 30 21:52:07.425580 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:07.425546 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" podUID="b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" containerName="maas-api" containerID="cri-o://12aecbc86354713528e9d1e73bdcb7fd4d83d3a2846be208e0ca947fee298ca2" gracePeriod=30 Oct 30 21:52:07.882974 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:07.872897 6808 generic.go:356] "Generic (PLEG): container finished" podID="b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" containerID="12aecbc86354713528e9d1e73bdcb7fd4d83d3a2846be208e0ca947fee298ca2" exitCode=0 Oct 30 21:52:07.883309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:07.883279 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" event={"ID":"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1","Type":"ContainerDied","Data":"12aecbc86354713528e9d1e73bdcb7fd4d83d3a2846be208e0ca947fee298ca2"} Oct 30 21:52:08.022231 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.022202 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:52:08.137490 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.137447 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls\") pod \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " Oct 30 21:52:08.137699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.137505 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs\") pod \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " Oct 30 21:52:08.137699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.137529 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9\" (UniqueName: \"kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9\") pod \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " Oct 30 21:52:08.137699 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.137598 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca\") pod \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\" (UID: \"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1\") " Oct 30 21:52:08.138106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.138076 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca" pod "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" (UID: "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:08.139612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.139583 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls" pod "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" (UID: "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:08.139725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.139621 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs" pod "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" (UID: "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:08.139725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.139670 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9" pod "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" (UID: "b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1"). InnerVolumeSpecName "kube-api-access-nt8k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:08.238959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.238904 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:08.238959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.238936 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:08.239190 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.238972 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:08.239190 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.238982 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8k9\" (UniqueName: \"kubernetes.io/projected/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1-kube-api-access-nt8k9\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:08.890357 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.890327 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" Oct 30 21:52:08.890808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.890356 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw" event={"ID":"b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1","Type":"ContainerDied","Data":"6453b500271cd75f5edf5f0646f685489578f991cf19c37eeba266e327fa7144"} Oct 30 21:52:08.890808 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.890421 6808 scope.go:122] "RemoveContainer" containerID="12aecbc86354713528e9d1e73bdcb7fd4d83d3a2846be208e0ca947fee298ca2" Oct 30 21:52:08.915466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.915441 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw"] Oct 30 21:52:08.924081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:08.924057 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-caa3dc-65cf76fb65-w8flw"] Oct 30 21:52:09.928797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:09.928762 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" path="/var/lib/kubelet/pods/b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1/volumes" Oct 30 21:52:16.032826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:16.032793 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6"] Oct 30 21:52:16.033742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:16.033703 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" containerID="cri-o://44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316" gracePeriod=30 Oct 30 21:52:16.036243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:16.034908 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t"] Oct 30 21:52:16.036243 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:16.035098 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" containerID="cri-o://b6a27811363b7287c9cfde2539e27f621e0604e32b02b3422df2577a83eff5c4" gracePeriod=30 Oct 30 21:52:16.698986 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:16.698937 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:16.699149 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:16.699006 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:20.367718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.367677 6808 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:20.368133 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.367751 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:20.615456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615414 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-gpnsf"] Oct 30 21:52:20.615871 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615853 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615874 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615889 6808 cpu_manager.go:414] "containerMap: removing container" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615896 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615908 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615916 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615937 6808 cpu_manager.go:414] "containerMap: removing container" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615944 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615973 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" Oct 30 21:52:20.615983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615980 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.615998 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" containerName="maas-api" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616005 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" containerName="maas-api" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616028 6808 cpu_manager.go:414] "containerMap: removing container" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616037 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616049 6808 cpu_manager.go:414] "containerMap: removing container" podUID="7dd84627-40a3-472e-8d9c-b1aa22437cd4" containerName="revoke-keys" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616056 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd84627-40a3-472e-8d9c-b1aa22437cd4" containerName="revoke-keys" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616070 6808 cpu_manager.go:414] "containerMap: removing container" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616078 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616089 6808 cpu_manager.go:414] "containerMap: removing container" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616096 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616106 6808 cpu_manager.go:414] "containerMap: removing container" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616113 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616205 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc26646-e9c2-4096-90ad-4b0df42ba0e1" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616218 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd84627-40a3-472e-8d9c-b1aa22437cd4" containerName="revoke-keys" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616227 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb0c9aa-72a0-40ab-a90f-869389caa158" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616237 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f39f5cc-0dcd-49b9-962b-9ea951a2dd32" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616248 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2439ab7-e15d-4bbf-b6b1-affdb57e26f8" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616258 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616268 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ffe4f8-d41c-4236-a34e-19a756febd79" containerName="main" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616277 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f3f5a4-1224-4297-91da-291efc553540" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616289 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6e6998-bee0-47aa-9fe6-87168a547b1e" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616298 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a3a86b-dbb8-4a28-ab1b-42b1289bd1c1" containerName="istio-proxy" Oct 30 21:52:20.616453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.616309 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b9d2e9-be36-448d-9ad7-cbeaab68a5c1" containerName="maas-api" Oct 30 21:52:20.619812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.619759 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.626570 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.626436 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4v6cn\"" Oct 30 21:52:20.627544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.627515 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:52:20.629791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.628099 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:52:20.630623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.628157 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:52:20.632725 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.632699 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-gpnsf"] Oct 30 21:52:20.674489 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.674385 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgjq\" (UniqueName: \"kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.674657 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.674538 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.674781 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.674758 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.776459 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.776422 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgjq\" (UniqueName: \"kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.776665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.776483 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.776665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.776586 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.779717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.779683 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.780747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.780718 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.789869 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.789837 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgjq\" (UniqueName: \"kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq\") pod \"maas-controller-7bff7d9b4d-gpnsf\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:20.939262 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:20.939225 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:21.237135 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.232834 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-gpnsf"] Oct 30 21:52:21.251169 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:52:21.251122 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f165e0_ad1b_4c88_8785_a58cd860796c.slice/crio-f04b2e4a77455de0f4c6667c88a0688263f6355a7a2f81a4c01054c4eba9d58f WatchSource:0}: Error finding container f04b2e4a77455de0f4c6667c88a0688263f6355a7a2f81a4c01054c4eba9d58f: Status 404 returned error can't find the container with id f04b2e4a77455de0f4c6667c88a0688263f6355a7a2f81a4c01054c4eba9d58f Oct 30 21:52:21.366105 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.365810 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512285 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512397 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512474 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512511 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512546 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512593 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512662 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512708 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.513394 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.512745 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert\") pod \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\" (UID: \"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1\") " Oct 30 21:52:21.514890 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.514530 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:21.516498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.516394 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.516498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.516469 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.516966 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.516916 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.517267 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.517248 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.529561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.529510 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:21.529561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.529516 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:21.532120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.532072 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "kube-api-access-j8zww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:21.532612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.532578 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy" pod "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" (UID: "a6f56aa7-dc0b-46df-89eb-eff7d729c6b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615372 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zww\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-kube-api-access-j8zww\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615411 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615427 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615441 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615455 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615471 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615485 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615499 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.615547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:21.615510 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.018976 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.018853 6808 generic.go:356] "Generic (PLEG): container finished" podID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerID="b6a27811363b7287c9cfde2539e27f621e0604e32b02b3422df2577a83eff5c4" exitCode=0 Oct 30 21:52:22.019316 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.019228 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" event={"ID":"cc153255-a9df-48c9-9f5a-dc57f2834c52","Type":"ContainerDied","Data":"b6a27811363b7287c9cfde2539e27f621e0604e32b02b3422df2577a83eff5c4"} Oct 30 21:52:22.020866 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.020839 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" event={"ID":"a0f165e0-ad1b-4c88-8785-a58cd860796c","Type":"ContainerStarted","Data":"f04b2e4a77455de0f4c6667c88a0688263f6355a7a2f81a4c01054c4eba9d58f"} Oct 30 21:52:22.048214 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.048180 6808 generic.go:356] "Generic (PLEG): container finished" podID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerID="44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316" exitCode=0 Oct 30 21:52:22.048482 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.048459 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" event={"ID":"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1","Type":"ContainerDied","Data":"44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316"} Oct 30 21:52:22.048637 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.048621 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" event={"ID":"a6f56aa7-dc0b-46df-89eb-eff7d729c6b1","Type":"ContainerDied","Data":"1aac186a74d7d81bea77e3b52694ed454fda7695dd69af4c3cdf7158ac818dfe"} Oct 30 21:52:22.048772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.048758 6808 scope.go:122] "RemoveContainer" containerID="44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316" Oct 30 21:52:22.049059 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.049040 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6" Oct 30 21:52:22.161828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.161749 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6"] Oct 30 21:52:22.180985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.177423 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-g8sd6"] Oct 30 21:52:22.338282 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.337502 6808 scope.go:122] "RemoveContainer" containerID="44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316" Oct 30 21:52:22.338282 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:52:22.338101 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316\": container with ID starting with 44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316 not found: ID does not exist" containerID="44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316" Oct 30 21:52:22.338282 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.338142 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316"} err="failed to get container status \"44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316\": rpc error: code = NotFound desc = could not find container \"44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316\": container with ID starting with 44f6bdc605ecac5d418cfee1beb8846a84485b72da621aeb221bf9f6505c0316 not found: ID does not exist" Oct 30 21:52:22.493792 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.493481 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590213 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590284 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590325 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590357 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590386 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590411 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590447 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590490 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.594055 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.590646 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert\") pod \"cc153255-a9df-48c9-9f5a-dc57f2834c52\" (UID: \"cc153255-a9df-48c9-9f5a-dc57f2834c52\") " Oct 30 21:52:22.606766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.597231 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:22.606766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.599269 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:22.606766 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.599369 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:22.607190 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.607150 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:22.607434 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.607412 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:22.637312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.637081 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:22.637312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.637159 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:22.637312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.637230 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:22.637312 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.637282 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx" pod "cc153255-a9df-48c9-9f5a-dc57f2834c52" (UID: "cc153255-a9df-48c9-9f5a-dc57f2834c52"). InnerVolumeSpecName "kube-api-access-vgbkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691591 6808 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-credential-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691625 6808 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-data\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691642 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbkx\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-kube-api-access-vgbkx\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691656 6808 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691670 6808 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-podinfo\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691700 6808 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-envoy\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691716 6808 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc153255-a9df-48c9-9f5a-dc57f2834c52-istio-token\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691731 6808 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc153255-a9df-48c9-9f5a-dc57f2834c52-workload-socket\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.691779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:22.691745 6808 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc153255-a9df-48c9-9f5a-dc57f2834c52-istiod-ca-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:23.075972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.075039 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" event={"ID":"cc153255-a9df-48c9-9f5a-dc57f2834c52","Type":"ContainerDied","Data":"98a37102a58a5bd0b31561ff2b6b6a171e9d339887872db4bc3104e13e69ecd5"} Oct 30 21:52:23.075972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.075110 6808 scope.go:122] "RemoveContainer" containerID="b6a27811363b7287c9cfde2539e27f621e0604e32b02b3422df2577a83eff5c4" Oct 30 21:52:23.075972 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.075223 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t" Oct 30 21:52:23.146520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.146456 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t"] Oct 30 21:52:23.155535 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.155509 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-fwd4t"] Oct 30 21:52:23.936150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.936091 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" path="/var/lib/kubelet/pods/a6f56aa7-dc0b-46df-89eb-eff7d729c6b1/volumes" Oct 30 21:52:23.937069 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:23.937037 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" path="/var/lib/kubelet/pods/cc153255-a9df-48c9-9f5a-dc57f2834c52/volumes" Oct 30 21:52:25.091579 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:25.091546 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" event={"ID":"a0f165e0-ad1b-4c88-8785-a58cd860796c","Type":"ContainerStarted","Data":"6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24"} Oct 30 21:52:25.091991 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:25.091631 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:25.115577 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:25.115526 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" podStartSLOduration=2.014826517 podStartE2EDuration="5.115513193s" totalImagesPullingTime="3.100686676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:25.1133209 +0000 UTC m=+1407.309012023" watchObservedRunningTime="2026-10-30 21:52:25.115513193 +0000 UTC m=+1407.311204309" Oct 30 21:52:35.697741 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:52:35.697704 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:52:36.098865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:36.098792 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:40.853518 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.853482 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr"] Oct 30 21:52:40.854068 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.854048 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" Oct 30 21:52:40.854163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.854070 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" Oct 30 21:52:40.854163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.854110 6808 cpu_manager.go:414] "containerMap: removing container" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" Oct 30 21:52:40.854163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.854118 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" Oct 30 21:52:40.854323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.854231 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f56aa7-dc0b-46df-89eb-eff7d729c6b1" containerName="istio-proxy" Oct 30 21:52:40.854323 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.854249 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc153255-a9df-48c9-9f5a-dc57f2834c52" containerName="istio-proxy" Oct 30 21:52:40.857163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.857141 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:40.866158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.866135 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:40.866410 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.866395 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:52:40.881445 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.881417 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr"] Oct 30 21:52:40.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.910040 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:52:40.910067 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.910056 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:52:40.964893 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.964851 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:40.965106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.964928 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6ps\" (UniqueName: \"kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:40.965106 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:40.964995 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.066444 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.066413 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.066622 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.066479 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6ps\" (UniqueName: \"kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.066622 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.066511 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.067249 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.067222 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.068785 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.068756 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.084580 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.084558 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6ps\" (UniqueName: \"kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps\") pod \"maas-api-revoke-sub-85237e45-fq9jr\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.170350 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.170322 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:41.447592 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:41.447456 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr"] Oct 30 21:52:41.450438 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:52:41.450406 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d441e2_4d81_4975_a2db_5ed9aba58e58.slice/crio-b4ca92f20e063845ded007f6f18a325205ff57fdf41d5ad864269aabc6c0b0d6 WatchSource:0}: Error finding container b4ca92f20e063845ded007f6f18a325205ff57fdf41d5ad864269aabc6c0b0d6: Status 404 returned error can't find the container with id b4ca92f20e063845ded007f6f18a325205ff57fdf41d5ad864269aabc6c0b0d6 Oct 30 21:52:42.176891 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:42.176856 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" event={"ID":"84d441e2-4d81-4975-a2db-5ed9aba58e58","Type":"ContainerStarted","Data":"60d0c9d0168b90a90a72c021ef952fb8dea4b7a0107a9f74d23932a8900568d4"} Oct 30 21:52:42.177429 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:42.176910 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" event={"ID":"84d441e2-4d81-4975-a2db-5ed9aba58e58","Type":"ContainerStarted","Data":"b4ca92f20e063845ded007f6f18a325205ff57fdf41d5ad864269aabc6c0b0d6"} Oct 30 21:52:42.202997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:42.202923 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" podStartSLOduration=2.202910706 podStartE2EDuration="2.202910706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:42.202065258 +0000 UTC m=+1424.397756376" watchObservedRunningTime="2026-10-30 21:52:42.202910706 +0000 UTC m=+1424.398601824" Oct 30 21:52:43.183750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:43.183718 6808 generic.go:356] "Generic (PLEG): container finished" podID="84d441e2-4d81-4975-a2db-5ed9aba58e58" containerID="60d0c9d0168b90a90a72c021ef952fb8dea4b7a0107a9f74d23932a8900568d4" exitCode=0 Oct 30 21:52:43.184231 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:43.183800 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" event={"ID":"84d441e2-4d81-4975-a2db-5ed9aba58e58","Type":"ContainerDied","Data":"60d0c9d0168b90a90a72c021ef952fb8dea4b7a0107a9f74d23932a8900568d4"} Oct 30 21:52:44.334884 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.334854 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:44.498791 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.498720 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca\") pod \"84d441e2-4d81-4975-a2db-5ed9aba58e58\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " Oct 30 21:52:44.498938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.498846 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps\" (UniqueName: \"kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps\") pod \"84d441e2-4d81-4975-a2db-5ed9aba58e58\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " Oct 30 21:52:44.498938 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.498870 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca\") pod \"84d441e2-4d81-4975-a2db-5ed9aba58e58\" (UID: \"84d441e2-4d81-4975-a2db-5ed9aba58e58\") " Oct 30 21:52:44.499168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.499132 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca" pod "84d441e2-4d81-4975-a2db-5ed9aba58e58" (UID: "84d441e2-4d81-4975-a2db-5ed9aba58e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:44.500980 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.500933 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca" pod "84d441e2-4d81-4975-a2db-5ed9aba58e58" (UID: "84d441e2-4d81-4975-a2db-5ed9aba58e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:44.501088 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.501069 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps" pod "84d441e2-4d81-4975-a2db-5ed9aba58e58" (UID: "84d441e2-4d81-4975-a2db-5ed9aba58e58"). InnerVolumeSpecName "kube-api-access-2m6ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:44.599717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.599691 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:44.599717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.599714 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6ps\" (UniqueName: \"kubernetes.io/projected/84d441e2-4d81-4975-a2db-5ed9aba58e58-kube-api-access-2m6ps\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:44.599850 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:44.599724 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d441e2-4d81-4975-a2db-5ed9aba58e58-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:45.194536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:45.194509 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" Oct 30 21:52:45.194536 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:45.194527 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr" event={"ID":"84d441e2-4d81-4975-a2db-5ed9aba58e58","Type":"ContainerDied","Data":"b4ca92f20e063845ded007f6f18a325205ff57fdf41d5ad864269aabc6c0b0d6"} Oct 30 21:52:45.194747 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:45.194570 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4ca92f20e063845ded007f6f18a325205ff57fdf41d5ad864269aabc6c0b0d6" Oct 30 21:52:52.688175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:52.688140 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-gpnsf"] Oct 30 21:52:52.688792 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:52.688348 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" podUID="a0f165e0-ad1b-4c88-8785-a58cd860796c" containerName="manager" containerID="cri-o://6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24" gracePeriod=30 Oct 30 21:52:52.923894 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:52.923871 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:53.074083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.074006 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq\" (UniqueName: \"kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq\") pod \"a0f165e0-ad1b-4c88-8785-a58cd860796c\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " Oct 30 21:52:53.074083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.074066 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs\") pod \"a0f165e0-ad1b-4c88-8785-a58cd860796c\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " Oct 30 21:52:53.074271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.074157 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert\") pod \"a0f165e0-ad1b-4c88-8785-a58cd860796c\" (UID: \"a0f165e0-ad1b-4c88-8785-a58cd860796c\") " Oct 30 21:52:53.076237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.076201 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert" pod "a0f165e0-ad1b-4c88-8785-a58cd860796c" (UID: "a0f165e0-ad1b-4c88-8785-a58cd860796c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:53.076237 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.076226 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs" pod "a0f165e0-ad1b-4c88-8785-a58cd860796c" (UID: "a0f165e0-ad1b-4c88-8785-a58cd860796c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:53.076449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.076231 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq" pod "a0f165e0-ad1b-4c88-8785-a58cd860796c" (UID: "a0f165e0-ad1b-4c88-8785-a58cd860796c"). InnerVolumeSpecName "kube-api-access-zbgjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:53.175119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.175093 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgjq\" (UniqueName: \"kubernetes.io/projected/a0f165e0-ad1b-4c88-8785-a58cd860796c-kube-api-access-zbgjq\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:53.175119 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.175117 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:53.175273 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.175128 6808 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f165e0-ad1b-4c88-8785-a58cd860796c-webhook-cert\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:53.236477 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.236436 6808 generic.go:356] "Generic (PLEG): container finished" podID="a0f165e0-ad1b-4c88-8785-a58cd860796c" containerID="6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24" exitCode=0 Oct 30 21:52:53.236633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.236500 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" Oct 30 21:52:53.236633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.236515 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" event={"ID":"a0f165e0-ad1b-4c88-8785-a58cd860796c","Type":"ContainerDied","Data":"6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24"} Oct 30 21:52:53.236633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.236597 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bff7d9b4d-gpnsf" event={"ID":"a0f165e0-ad1b-4c88-8785-a58cd860796c","Type":"ContainerDied","Data":"f04b2e4a77455de0f4c6667c88a0688263f6355a7a2f81a4c01054c4eba9d58f"} Oct 30 21:52:53.236633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.236620 6808 scope.go:122] "RemoveContainer" containerID="6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24" Oct 30 21:52:53.245662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.245645 6808 scope.go:122] "RemoveContainer" containerID="6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24" Oct 30 21:52:53.245978 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:52:53.245942 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24\": container with ID starting with 6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24 not found: ID does not exist" containerID="6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24" Oct 30 21:52:53.246034 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.245987 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24"} err="failed to get container status \"6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24\": rpc error: code = NotFound desc = could not find container \"6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24\": container with ID starting with 6245e891e9f0044e6701b0bfdff7a7cb68e762860f79e1304027563e133fbc24 not found: ID does not exist" Oct 30 21:52:53.264216 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.264188 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-gpnsf"] Oct 30 21:52:53.269038 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.269019 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-gpnsf"] Oct 30 21:52:53.927334 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:53.927306 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f165e0-ad1b-4c88-8785-a58cd860796c" path="/var/lib/kubelet/pods/a0f165e0-ad1b-4c88-8785-a58cd860796c/volumes" Oct 30 21:52:54.521772 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.521737 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-4ftf4"] Oct 30 21:52:54.522112 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.522098 6808 cpu_manager.go:414] "containerMap: removing container" podUID="84d441e2-4d81-4975-a2db-5ed9aba58e58" containerName="revoke-keys" Oct 30 21:52:54.522166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.522113 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d441e2-4d81-4975-a2db-5ed9aba58e58" containerName="revoke-keys" Oct 30 21:52:54.522166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.522132 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a0f165e0-ad1b-4c88-8785-a58cd860796c" containerName="manager" Oct 30 21:52:54.522166 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.522137 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f165e0-ad1b-4c88-8785-a58cd860796c" containerName="manager" Oct 30 21:52:54.522260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.522204 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d441e2-4d81-4975-a2db-5ed9aba58e58" containerName="revoke-keys" Oct 30 21:52:54.522260 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.522214 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f165e0-ad1b-4c88-8785-a58cd860796c" containerName="manager" Oct 30 21:52:54.526465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.526447 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.533286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.533261 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:52:54.533404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.533300 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:52:54.533404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.533337 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:52:54.533404 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.533386 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4v6cn\"" Oct 30 21:52:54.540120 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.540094 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-4ftf4"] Oct 30 21:52:54.685690 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.685662 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10aa493a-209e-4052-bed7-47319c308928-metrics-certs\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.685865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.685776 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlth\" (UniqueName: \"kubernetes.io/projected/10aa493a-209e-4052-bed7-47319c308928-kube-api-access-wjlth\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.685935 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.685905 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10aa493a-209e-4052-bed7-47319c308928-webhook-cert\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.786854 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.786793 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10aa493a-209e-4052-bed7-47319c308928-webhook-cert\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.786854 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.786842 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10aa493a-209e-4052-bed7-47319c308928-metrics-certs\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.787044 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.786877 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlth\" (UniqueName: \"kubernetes.io/projected/10aa493a-209e-4052-bed7-47319c308928-kube-api-access-wjlth\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.789292 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.789268 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10aa493a-209e-4052-bed7-47319c308928-webhook-cert\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.789389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.789326 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10aa493a-209e-4052-bed7-47319c308928-metrics-certs\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.809084 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.809065 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlth\" (UniqueName: \"kubernetes.io/projected/10aa493a-209e-4052-bed7-47319c308928-kube-api-access-wjlth\") pod \"maas-controller-7bff7d9b4d-4ftf4\" (UID: \"10aa493a-209e-4052-bed7-47319c308928\") " pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:54.839499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:54.839475 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:55.116107 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:55.116082 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bff7d9b4d-4ftf4"] Oct 30 21:52:55.118588 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:52:55.118561 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10aa493a_209e_4052_bed7_47319c308928.slice/crio-04fd85baf5a8847f4bfaa725f9d54a7c2ad3b2b77cf68c8e406f3b3305254268 WatchSource:0}: Error finding container 04fd85baf5a8847f4bfaa725f9d54a7c2ad3b2b77cf68c8e406f3b3305254268: Status 404 returned error can't find the container with id 04fd85baf5a8847f4bfaa725f9d54a7c2ad3b2b77cf68c8e406f3b3305254268 Oct 30 21:52:55.248815 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:55.248778 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" event={"ID":"10aa493a-209e-4052-bed7-47319c308928","Type":"ContainerStarted","Data":"04fd85baf5a8847f4bfaa725f9d54a7c2ad3b2b77cf68c8e406f3b3305254268"} Oct 30 21:52:56.255919 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:56.255883 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" event={"ID":"10aa493a-209e-4052-bed7-47319c308928","Type":"ContainerStarted","Data":"ae2b2c84a53b0551e64faa86702624d5e1825bf484b1788f9402ade11bdfa915"} Oct 30 21:52:56.256363 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:56.255985 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:52:56.279495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:52:56.279449 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" podStartSLOduration=1.792269048 podStartE2EDuration="2.279434734s" totalImagesPullingTime="487.165686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:56.277535447 +0000 UTC m=+1438.473226565" watchObservedRunningTime="2026-10-30 21:52:56.279434734 +0000 UTC m=+1438.475125853" Oct 30 21:53:05.697559 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:53:05.697520 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:53:07.263605 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:07.263580 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bff7d9b4d-4ftf4" Oct 30 21:53:34.048019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.047986 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw"] Oct 30 21:53:34.051701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.051684 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.057659 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.057630 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:34.057750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.057666 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:53:34.060452 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.060434 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:34.061049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.061034 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:34.090168 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.090148 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw"] Oct 30 21:53:34.153449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.153418 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5mh\" (UniqueName: \"kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.153573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.153471 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.153573 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.153548 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.254958 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.254931 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5mh\" (UniqueName: \"kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.255061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.254994 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.255061 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.255024 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.255560 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.255544 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.257288 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.257270 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.268101 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.268078 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5mh\" (UniqueName: \"kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh\") pod \"maas-api-revoke-sub-c1e271a9-jjncw\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.364258 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.364209 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:34.645402 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:34.645370 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw"] Oct 30 21:53:34.648499 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:53:34.648474 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ff3943_5068_4e2b_b26a_f455d06f6c5a.slice/crio-d03186d207b8d600ad1379a6d5442d547057bed8f9fa647678a7c02ccbd88ee6 WatchSource:0}: Error finding container d03186d207b8d600ad1379a6d5442d547057bed8f9fa647678a7c02ccbd88ee6: Status 404 returned error can't find the container with id d03186d207b8d600ad1379a6d5442d547057bed8f9fa647678a7c02ccbd88ee6 Oct 30 21:53:35.447361 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:35.447331 6808 generic.go:356] "Generic (PLEG): container finished" podID="76ff3943-5068-4e2b-b26a-f455d06f6c5a" containerID="e1d425f02001224649e38901f9423b00318db34b7b056d53e07061009230ff39" exitCode=0 Oct 30 21:53:35.447742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:35.447374 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" event={"ID":"76ff3943-5068-4e2b-b26a-f455d06f6c5a","Type":"ContainerDied","Data":"e1d425f02001224649e38901f9423b00318db34b7b056d53e07061009230ff39"} Oct 30 21:53:35.447742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:35.447421 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" event={"ID":"76ff3943-5068-4e2b-b26a-f455d06f6c5a","Type":"ContainerStarted","Data":"d03186d207b8d600ad1379a6d5442d547057bed8f9fa647678a7c02ccbd88ee6"} Oct 30 21:53:35.697563 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:53:35.697533 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:53:36.578057 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.578035 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:36.675520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.675495 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh\" (UniqueName: \"kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh\") pod \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " Oct 30 21:53:36.675694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.675541 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca\") pod \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " Oct 30 21:53:36.675694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.675582 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca\") pod \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\" (UID: \"76ff3943-5068-4e2b-b26a-f455d06f6c5a\") " Oct 30 21:53:36.676021 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.676000 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca" pod "76ff3943-5068-4e2b-b26a-f455d06f6c5a" (UID: "76ff3943-5068-4e2b-b26a-f455d06f6c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:36.677629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.677604 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca" pod "76ff3943-5068-4e2b-b26a-f455d06f6c5a" (UID: "76ff3943-5068-4e2b-b26a-f455d06f6c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:36.677750 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.677635 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh" pod "76ff3943-5068-4e2b-b26a-f455d06f6c5a" (UID: "76ff3943-5068-4e2b-b26a-f455d06f6c5a"). InnerVolumeSpecName "kube-api-access-lz5mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:36.777030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.776976 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz5mh\" (UniqueName: \"kubernetes.io/projected/76ff3943-5068-4e2b-b26a-f455d06f6c5a-kube-api-access-lz5mh\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:36.777030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.776998 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:36.777030 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:36.777008 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ff3943-5068-4e2b-b26a-f455d06f6c5a-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:37.457779 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:37.457741 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" Oct 30 21:53:37.457983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:37.457781 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw" event={"ID":"76ff3943-5068-4e2b-b26a-f455d06f6c5a","Type":"ContainerDied","Data":"d03186d207b8d600ad1379a6d5442d547057bed8f9fa647678a7c02ccbd88ee6"} Oct 30 21:53:37.457983 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:37.457835 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d03186d207b8d600ad1379a6d5442d547057bed8f9fa647678a7c02ccbd88ee6" Oct 30 21:53:49.773146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:49.773095 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:53:49.773663 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:49.773179 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:52.885800 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.885756 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd"] Oct 30 21:53:52.886411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.886159 6808 cpu_manager.go:414] "containerMap: removing container" podUID="76ff3943-5068-4e2b-b26a-f455d06f6c5a" containerName="revoke-keys" Oct 30 21:53:52.886411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.886171 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ff3943-5068-4e2b-b26a-f455d06f6c5a" containerName="revoke-keys" Oct 30 21:53:52.886411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.886235 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ff3943-5068-4e2b-b26a-f455d06f6c5a" containerName="revoke-keys" Oct 30 21:53:52.889297 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.889277 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:52.902753 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.902727 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd"] Oct 30 21:53:52.904919 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.904870 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:52.905081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.905038 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:52.905173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.905087 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:52.906385 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.906363 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:53:52.945433 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.945396 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6fr\" (UniqueName: \"kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:52.945613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.945439 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:52.945613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:52.945519 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.046814 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.046780 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6fr\" (UniqueName: \"kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.047013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.046825 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.047013 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.046875 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.047681 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.047654 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.049367 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.049344 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.059940 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.059919 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6fr\" (UniqueName: \"kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr\") pod \"maas-api-revoke-sub-2e1f5b5f-bcpzd\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.208693 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.208611 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:53.485594 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.485511 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd"] Oct 30 21:53:53.488363 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:53:53.488319 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a8452c_ac08_4ae1_8a6e_097907b016d8.slice/crio-ce7b2c924e4d97816076d2698f8183a6ab118ffe0e8daa6a5d904b15a5b2c1ab WatchSource:0}: Error finding container ce7b2c924e4d97816076d2698f8183a6ab118ffe0e8daa6a5d904b15a5b2c1ab: Status 404 returned error can't find the container with id ce7b2c924e4d97816076d2698f8183a6ab118ffe0e8daa6a5d904b15a5b2c1ab Oct 30 21:53:53.551828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:53.551801 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" event={"ID":"e8a8452c-ac08-4ae1-8a6e-097907b016d8","Type":"ContainerStarted","Data":"ce7b2c924e4d97816076d2698f8183a6ab118ffe0e8daa6a5d904b15a5b2c1ab"} Oct 30 21:53:54.558127 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:54.558043 6808 generic.go:356] "Generic (PLEG): container finished" podID="e8a8452c-ac08-4ae1-8a6e-097907b016d8" containerID="714e87487774ec343281f0281255b2273677d4e37fa9488f7c21b7ba9c4e57cd" exitCode=0 Oct 30 21:53:54.558484 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:54.558134 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" event={"ID":"e8a8452c-ac08-4ae1-8a6e-097907b016d8","Type":"ContainerDied","Data":"714e87487774ec343281f0281255b2273677d4e37fa9488f7c21b7ba9c4e57cd"} Oct 30 21:53:55.693862 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.693840 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:55.773497 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.773471 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca\") pod \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " Oct 30 21:53:55.773626 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.773540 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca\") pod \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " Oct 30 21:53:55.773727 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.773704 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr\" (UniqueName: \"kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr\") pod \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\" (UID: \"e8a8452c-ac08-4ae1-8a6e-097907b016d8\") " Oct 30 21:53:55.773926 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.773904 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca" pod "e8a8452c-ac08-4ae1-8a6e-097907b016d8" (UID: "e8a8452c-ac08-4ae1-8a6e-097907b016d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:55.774112 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.774088 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.775400 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.775376 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca" pod "e8a8452c-ac08-4ae1-8a6e-097907b016d8" (UID: "e8a8452c-ac08-4ae1-8a6e-097907b016d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:55.775717 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.775701 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr" pod "e8a8452c-ac08-4ae1-8a6e-097907b016d8" (UID: "e8a8452c-ac08-4ae1-8a6e-097907b016d8"). InnerVolumeSpecName "kube-api-access-wd6fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:55.875530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.875471 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd6fr\" (UniqueName: \"kubernetes.io/projected/e8a8452c-ac08-4ae1-8a6e-097907b016d8-kube-api-access-wd6fr\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.875530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:55.875498 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a8452c-ac08-4ae1-8a6e-097907b016d8-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:56.568426 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:56.568399 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" Oct 30 21:53:56.568614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:56.568434 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd" event={"ID":"e8a8452c-ac08-4ae1-8a6e-097907b016d8","Type":"ContainerDied","Data":"ce7b2c924e4d97816076d2698f8183a6ab118ffe0e8daa6a5d904b15a5b2c1ab"} Oct 30 21:53:56.568614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:53:56.568482 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7b2c924e4d97816076d2698f8183a6ab118ffe0e8daa6a5d904b15a5b2c1ab" Oct 30 21:54:03.967181 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.967150 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4"] Oct 30 21:54:03.967558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.967516 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e8a8452c-ac08-4ae1-8a6e-097907b016d8" containerName="revoke-keys" Oct 30 21:54:03.967558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.967529 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a8452c-ac08-4ae1-8a6e-097907b016d8" containerName="revoke-keys" Oct 30 21:54:03.967635 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.967608 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a8452c-ac08-4ae1-8a6e-097907b016d8" containerName="revoke-keys" Oct 30 21:54:03.970990 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.970937 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:03.981466 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.981434 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:03.981613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.981498 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:03.981613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.981574 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:03.981613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.981582 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:54:03.988529 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:03.988502 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4"] Oct 30 21:54:04.048359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.048332 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.048555 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.048371 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.048555 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.048433 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbtd\" (UniqueName: \"kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.148995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.148930 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.149150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.149024 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.149150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.149073 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbtd\" (UniqueName: \"kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.149602 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.149580 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.151558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.151532 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.177090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.177057 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbtd\" (UniqueName: \"kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd\") pod \"maas-api-revoke-sub-ec843018-hhgn4\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.286562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.286477 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:04.567883 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.567808 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4"] Oct 30 21:54:04.571746 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:54:04.571717 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3817c72e_120c_47e6_8853_7b3166117104.slice/crio-bfe665585a021132d6ceb4ff0859628d08bad1b570919feaff3bdaebb1abf944 WatchSource:0}: Error finding container bfe665585a021132d6ceb4ff0859628d08bad1b570919feaff3bdaebb1abf944: Status 404 returned error can't find the container with id bfe665585a021132d6ceb4ff0859628d08bad1b570919feaff3bdaebb1abf944 Oct 30 21:54:04.606356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.606321 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" event={"ID":"3817c72e-120c-47e6-8853-7b3166117104","Type":"ContainerStarted","Data":"bfe665585a021132d6ceb4ff0859628d08bad1b570919feaff3bdaebb1abf944"} Oct 30 21:54:04.904100 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:04.904072 6808 scope.go:122] "RemoveContainer" containerID="84075b4f6cdf0bb7039c12ae5b51bce0f07eddc30119a35501c59d9635b9d985" Oct 30 21:54:05.612449 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:05.612372 6808 generic.go:356] "Generic (PLEG): container finished" podID="3817c72e-120c-47e6-8853-7b3166117104" containerID="0a193cd78ae9c88677be5aa4f632d202afe3039afbe4538a9d864ff5429823ed" exitCode=0 Oct 30 21:54:05.612846 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:05.612461 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" event={"ID":"3817c72e-120c-47e6-8853-7b3166117104","Type":"ContainerDied","Data":"0a193cd78ae9c88677be5aa4f632d202afe3039afbe4538a9d864ff5429823ed"} Oct 30 21:54:05.699077 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:54:05.699042 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:54:06.771860 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.771829 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:06.881389 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.881352 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca\") pod \"3817c72e-120c-47e6-8853-7b3166117104\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " Oct 30 21:54:06.881584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.881461 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd\" (UniqueName: \"kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd\") pod \"3817c72e-120c-47e6-8853-7b3166117104\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " Oct 30 21:54:06.881584 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.881557 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca\") pod \"3817c72e-120c-47e6-8853-7b3166117104\" (UID: \"3817c72e-120c-47e6-8853-7b3166117104\") " Oct 30 21:54:06.882187 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.882156 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca" pod "3817c72e-120c-47e6-8853-7b3166117104" (UID: "3817c72e-120c-47e6-8853-7b3166117104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:06.884259 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.884127 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd" pod "3817c72e-120c-47e6-8853-7b3166117104" (UID: "3817c72e-120c-47e6-8853-7b3166117104"). InnerVolumeSpecName "kube-api-access-gsbtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:06.884393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.884302 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca" pod "3817c72e-120c-47e6-8853-7b3166117104" (UID: "3817c72e-120c-47e6-8853-7b3166117104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:06.985706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.985601 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3817c72e-120c-47e6-8853-7b3166117104-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.985706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.985643 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsbtd\" (UniqueName: \"kubernetes.io/projected/3817c72e-120c-47e6-8853-7b3166117104-kube-api-access-gsbtd\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.985706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:06.985661 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3817c72e-120c-47e6-8853-7b3166117104-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:07.632232 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:07.632200 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" Oct 30 21:54:07.632596 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:07.632200 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4" event={"ID":"3817c72e-120c-47e6-8853-7b3166117104","Type":"ContainerDied","Data":"bfe665585a021132d6ceb4ff0859628d08bad1b570919feaff3bdaebb1abf944"} Oct 30 21:54:07.632596 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:07.632566 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe665585a021132d6ceb4ff0859628d08bad1b570919feaff3bdaebb1abf944" Oct 30 21:54:19.779121 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:19.779078 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:54:19.779801 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:19.779767 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:30.390666 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.390630 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92"] Oct 30 21:54:30.391256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.391236 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3817c72e-120c-47e6-8853-7b3166117104" containerName="revoke-keys" Oct 30 21:54:30.391347 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.391258 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3817c72e-120c-47e6-8853-7b3166117104" containerName="revoke-keys" Oct 30 21:54:30.391408 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.391364 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3817c72e-120c-47e6-8853-7b3166117104" containerName="revoke-keys" Oct 30 21:54:30.394689 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.394669 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.402587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.402560 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:30.402587 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.402583 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:54:30.404534 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.404492 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92"] Oct 30 21:54:30.404672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.404631 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:30.404741 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.404677 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:30.436729 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.436699 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmhb\" (UniqueName: \"kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.436853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.436760 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.436930 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.436855 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.538020 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.537993 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.538130 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.538080 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmhb\" (UniqueName: \"kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.538206 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.538190 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.538832 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.538814 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.540381 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.540360 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.550959 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.550914 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmhb\" (UniqueName: \"kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb\") pod \"maas-api-revoke-sub-046e1343-98r92\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.708393 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.708332 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:30.991094 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:30.991013 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92"] Oct 30 21:54:30.994620 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:54:30.994595 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426acc21_44a4_47c1_93dc_786b0ea894bd.slice/crio-0b36876ee92ab4d2a5a5ac48adde708acdd79e1d5d13ba68c68767228b955a2b WatchSource:0}: Error finding container 0b36876ee92ab4d2a5a5ac48adde708acdd79e1d5d13ba68c68767228b955a2b: Status 404 returned error can't find the container with id 0b36876ee92ab4d2a5a5ac48adde708acdd79e1d5d13ba68c68767228b955a2b Oct 30 21:54:31.762547 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:31.762520 6808 generic.go:356] "Generic (PLEG): container finished" podID="426acc21-44a4-47c1-93dc-786b0ea894bd" containerID="0c11b8527c9ae36eaece8b3c4c1af39847e3e7e272c0cf5c02d43901e759acb8" exitCode=0 Oct 30 21:54:31.762853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:31.762589 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" event={"ID":"426acc21-44a4-47c1-93dc-786b0ea894bd","Type":"ContainerDied","Data":"0c11b8527c9ae36eaece8b3c4c1af39847e3e7e272c0cf5c02d43901e759acb8"} Oct 30 21:54:31.762853 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:31.762625 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" event={"ID":"426acc21-44a4-47c1-93dc-786b0ea894bd","Type":"ContainerStarted","Data":"0b36876ee92ab4d2a5a5ac48adde708acdd79e1d5d13ba68c68767228b955a2b"} Oct 30 21:54:32.890532 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.890509 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:32.957287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.957260 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca\") pod \"426acc21-44a4-47c1-93dc-786b0ea894bd\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " Oct 30 21:54:32.957422 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.957298 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb\" (UniqueName: \"kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb\") pod \"426acc21-44a4-47c1-93dc-786b0ea894bd\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " Oct 30 21:54:32.957422 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.957333 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca\") pod \"426acc21-44a4-47c1-93dc-786b0ea894bd\" (UID: \"426acc21-44a4-47c1-93dc-786b0ea894bd\") " Oct 30 21:54:32.957649 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.957629 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca" pod "426acc21-44a4-47c1-93dc-786b0ea894bd" (UID: "426acc21-44a4-47c1-93dc-786b0ea894bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:32.959378 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.959346 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb" pod "426acc21-44a4-47c1-93dc-786b0ea894bd" (UID: "426acc21-44a4-47c1-93dc-786b0ea894bd"). InnerVolumeSpecName "kube-api-access-lrmhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:32.959486 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:32.959396 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca" pod "426acc21-44a4-47c1-93dc-786b0ea894bd" (UID: "426acc21-44a4-47c1-93dc-786b0ea894bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:33.058018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:33.057943 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:33.058018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:33.057985 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmhb\" (UniqueName: \"kubernetes.io/projected/426acc21-44a4-47c1-93dc-786b0ea894bd-kube-api-access-lrmhb\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:33.058018 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:33.057998 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426acc21-44a4-47c1-93dc-786b0ea894bd-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:33.773162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:33.773132 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" Oct 30 21:54:33.773311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:33.773139 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92" event={"ID":"426acc21-44a4-47c1-93dc-786b0ea894bd","Type":"ContainerDied","Data":"0b36876ee92ab4d2a5a5ac48adde708acdd79e1d5d13ba68c68767228b955a2b"} Oct 30 21:54:33.773311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:33.773265 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b36876ee92ab4d2a5a5ac48adde708acdd79e1d5d13ba68c68767228b955a2b" Oct 30 21:54:35.698231 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:54:35.698192 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:54:40.667173 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.667095 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt"] Oct 30 21:54:40.667614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.667505 6808 cpu_manager.go:414] "containerMap: removing container" podUID="426acc21-44a4-47c1-93dc-786b0ea894bd" containerName="revoke-keys" Oct 30 21:54:40.667614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.667518 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426acc21-44a4-47c1-93dc-786b0ea894bd" containerName="revoke-keys" Oct 30 21:54:40.667614 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.667592 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="426acc21-44a4-47c1-93dc-786b0ea894bd" containerName="revoke-keys" Oct 30 21:54:40.670701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.670685 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.676620 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.676599 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:40.677937 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.677919 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:54:40.683461 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.683435 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt"] Oct 30 21:54:40.721365 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.721343 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:40.721498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.721473 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:40.726912 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.726889 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.727027 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.726975 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvct\" (UniqueName: \"kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.727079 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.727025 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.827865 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.827832 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.828039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.827915 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvct\" (UniqueName: \"kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.828039 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.827974 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.828633 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.828608 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.830198 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.830178 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.839960 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.839923 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvct\" (UniqueName: \"kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct\") pod \"maas-api-revoke-sub-d3fba056-xz2rt\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:40.983829 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:40.983751 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:41.268844 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:41.268820 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt"] Oct 30 21:54:41.271180 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:54:41.271143 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bfd8f6_29a6_4ab6_827e_11ce4472d85a.slice/crio-720b6496bb4f0f25d67edd93098e50af178f14f4a1a97f7d4541ad657586af64 WatchSource:0}: Error finding container 720b6496bb4f0f25d67edd93098e50af178f14f4a1a97f7d4541ad657586af64: Status 404 returned error can't find the container with id 720b6496bb4f0f25d67edd93098e50af178f14f4a1a97f7d4541ad657586af64 Oct 30 21:54:41.812887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:41.812845 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" event={"ID":"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a","Type":"ContainerStarted","Data":"c43b69e6023c513e4f825913a46063bf47eeb21ca14a99f80d999c70638662a7"} Oct 30 21:54:41.813321 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:41.812897 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" event={"ID":"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a","Type":"ContainerStarted","Data":"720b6496bb4f0f25d67edd93098e50af178f14f4a1a97f7d4541ad657586af64"} Oct 30 21:54:41.835089 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:41.835037 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" podStartSLOduration=1.835024671 podStartE2EDuration="1.835024671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:41.832576085 +0000 UTC m=+1544.028267206" watchObservedRunningTime="2026-10-30 21:54:41.835024671 +0000 UTC m=+1544.030715787" Oct 30 21:54:42.840973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:42.840187 6808 generic.go:356] "Generic (PLEG): container finished" podID="e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" containerID="c43b69e6023c513e4f825913a46063bf47eeb21ca14a99f80d999c70638662a7" exitCode=0 Oct 30 21:54:42.840973 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:42.840282 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" event={"ID":"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a","Type":"ContainerDied","Data":"c43b69e6023c513e4f825913a46063bf47eeb21ca14a99f80d999c70638662a7"} Oct 30 21:54:44.242564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.242537 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:44.338340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.338296 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca\") pod \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " Oct 30 21:54:44.338520 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.338438 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct\" (UniqueName: \"kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct\") pod \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " Oct 30 21:54:44.338589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.338542 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca\") pod \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\" (UID: \"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a\") " Oct 30 21:54:44.339203 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.339175 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca" pod "e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" (UID: "e9bfd8f6-29a6-4ab6-827e-11ce4472d85a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:44.344205 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.344180 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct" pod "e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" (UID: "e9bfd8f6-29a6-4ab6-827e-11ce4472d85a"). InnerVolumeSpecName "kube-api-access-4nvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:44.375551 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.375523 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca" pod "e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" (UID: "e9bfd8f6-29a6-4ab6-827e-11ce4472d85a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:44.439701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.439670 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvct\" (UniqueName: \"kubernetes.io/projected/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-kube-api-access-4nvct\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.439701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.439696 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.439701 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.439706 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:44.860624 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.860595 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" Oct 30 21:54:44.860794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.860602 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt" event={"ID":"e9bfd8f6-29a6-4ab6-827e-11ce4472d85a","Type":"ContainerDied","Data":"720b6496bb4f0f25d67edd93098e50af178f14f4a1a97f7d4541ad657586af64"} Oct 30 21:54:44.860794 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:54:44.860715 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="720b6496bb4f0f25d67edd93098e50af178f14f4a1a97f7d4541ad657586af64" Oct 30 21:55:01.695333 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.695298 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl"] Oct 30 21:55:01.695702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.695660 6808 cpu_manager.go:414] "containerMap: removing container" podUID="e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" containerName="revoke-keys" Oct 30 21:55:01.695702 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.695670 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" containerName="revoke-keys" Oct 30 21:55:01.695778 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.695748 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" containerName="revoke-keys" Oct 30 21:55:01.699098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.699068 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.704868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.704842 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:55:01.704868 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.704865 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:01.705098 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.704887 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:01.710670 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.710652 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:01.719910 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.719886 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl"] Oct 30 21:55:01.845204 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.845175 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.845364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.845214 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.845364 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.845332 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwld\" (UniqueName: \"kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.946703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.946631 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwld\" (UniqueName: \"kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.946703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.946697 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.946881 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.946734 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.947286 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.947258 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.949090 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.949060 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:01.960755 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:01.960735 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwld\" (UniqueName: \"kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld\") pod \"maas-api-revoke-sub-ae594979-96dsl\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:02.012875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:02.012852 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:02.296454 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:02.296388 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl"] Oct 30 21:55:02.300887 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:55:02.300854 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8137d0a1_3b34_42f2_97bd_77b8e24b26bb.slice/crio-29943921c49acea33410569bfbfee279c37118753e2bda39d2aa1fe00f68226e WatchSource:0}: Error finding container 29943921c49acea33410569bfbfee279c37118753e2bda39d2aa1fe00f68226e: Status 404 returned error can't find the container with id 29943921c49acea33410569bfbfee279c37118753e2bda39d2aa1fe00f68226e Oct 30 21:55:02.951612 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:02.951571 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" event={"ID":"8137d0a1-3b34-42f2-97bd-77b8e24b26bb","Type":"ContainerStarted","Data":"d53c60f83b298dee8991b7d54e8d260c13d8e66dedbc216ef77efe35fb9c6ec3"} Oct 30 21:55:02.951997 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:02.951642 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" event={"ID":"8137d0a1-3b34-42f2-97bd-77b8e24b26bb","Type":"ContainerStarted","Data":"29943921c49acea33410569bfbfee279c37118753e2bda39d2aa1fe00f68226e"} Oct 30 21:55:02.984162 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:02.984105 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" podStartSLOduration=1.984088606 podStartE2EDuration="1.984088606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:02.981156729 +0000 UTC m=+1565.176847848" watchObservedRunningTime="2026-10-30 21:55:02.984088606 +0000 UTC m=+1565.179779737" Oct 30 21:55:03.958235 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:03.958200 6808 generic.go:356] "Generic (PLEG): container finished" podID="8137d0a1-3b34-42f2-97bd-77b8e24b26bb" containerID="d53c60f83b298dee8991b7d54e8d260c13d8e66dedbc216ef77efe35fb9c6ec3" exitCode=0 Oct 30 21:55:03.958637 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:03.958283 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" event={"ID":"8137d0a1-3b34-42f2-97bd-77b8e24b26bb","Type":"ContainerDied","Data":"d53c60f83b298dee8991b7d54e8d260c13d8e66dedbc216ef77efe35fb9c6ec3"} Oct 30 21:55:05.097616 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.097591 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:05.171734 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.171703 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld\" (UniqueName: \"kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld\") pod \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " Oct 30 21:55:05.171897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.171822 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca\") pod \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " Oct 30 21:55:05.171897 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.171862 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca\") pod \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\" (UID: \"8137d0a1-3b34-42f2-97bd-77b8e24b26bb\") " Oct 30 21:55:05.172225 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.172202 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca" pod "8137d0a1-3b34-42f2-97bd-77b8e24b26bb" (UID: "8137d0a1-3b34-42f2-97bd-77b8e24b26bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:05.173833 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.173809 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca" pod "8137d0a1-3b34-42f2-97bd-77b8e24b26bb" (UID: "8137d0a1-3b34-42f2-97bd-77b8e24b26bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:05.174019 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.173819 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld" pod "8137d0a1-3b34-42f2-97bd-77b8e24b26bb" (UID: "8137d0a1-3b34-42f2-97bd-77b8e24b26bb"). InnerVolumeSpecName "kube-api-access-7lwld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:05.274537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.274463 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwld\" (UniqueName: \"kubernetes.io/projected/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-kube-api-access-7lwld\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:05.274537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.274492 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:05.274537 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.274502 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137d0a1-3b34-42f2-97bd-77b8e24b26bb-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:05.697967 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:55:05.697912 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:55:05.968745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.968664 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" Oct 30 21:55:05.968745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.968689 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl" event={"ID":"8137d0a1-3b34-42f2-97bd-77b8e24b26bb","Type":"ContainerDied","Data":"29943921c49acea33410569bfbfee279c37118753e2bda39d2aa1fe00f68226e"} Oct 30 21:55:05.968745 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:05.968733 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29943921c49acea33410569bfbfee279c37118753e2bda39d2aa1fe00f68226e" Oct 30 21:55:20.018415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.018378 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz"] Oct 30 21:55:20.018882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.018775 6808 cpu_manager.go:414] "containerMap: removing container" podUID="8137d0a1-3b34-42f2-97bd-77b8e24b26bb" containerName="revoke-keys" Oct 30 21:55:20.018882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.018791 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8137d0a1-3b34-42f2-97bd-77b8e24b26bb" containerName="revoke-keys" Oct 30 21:55:20.018882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.018864 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="8137d0a1-3b34-42f2-97bd-77b8e24b26bb" containerName="revoke-keys" Oct 30 21:55:20.021932 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.021917 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.030724 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.030701 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:20.031900 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.031880 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:20.032155 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.032135 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:20.032256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.032197 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:55:20.034993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.034940 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz"] Oct 30 21:55:20.039303 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.039281 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.039411 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.039345 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwf4f\" (UniqueName: \"kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.039560 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.039536 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.140821 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.140785 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwf4f\" (UniqueName: \"kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.140994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.140880 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.140994 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.140945 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.141499 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.141472 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.143213 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.143190 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.153553 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.153529 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwf4f\" (UniqueName: \"kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f\") pod \"maas-api-revoke-sub-a4aef7d7-7rzgz\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.334773 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.334707 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:20.624465 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:20.624442 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz"] Oct 30 21:55:20.626464 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:55:20.626430 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edf3409_61c0_4c08_8e7f_c179a40ccd59.slice/crio-1af4c5b0b76d831ebcdd0a4e0a3881de3c7605bc645024ac54672656ac77c104 WatchSource:0}: Error finding container 1af4c5b0b76d831ebcdd0a4e0a3881de3c7605bc645024ac54672656ac77c104: Status 404 returned error can't find the container with id 1af4c5b0b76d831ebcdd0a4e0a3881de3c7605bc645024ac54672656ac77c104 Oct 30 21:55:21.062480 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:21.062447 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" event={"ID":"3edf3409-61c0-4c08-8e7f-c179a40ccd59","Type":"ContainerStarted","Data":"08483574869c9db787450ec7c490ce096307a4582db12a6b20ae7682ad191690"} Oct 30 21:55:21.062807 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:21.062500 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" event={"ID":"3edf3409-61c0-4c08-8e7f-c179a40ccd59","Type":"ContainerStarted","Data":"1af4c5b0b76d831ebcdd0a4e0a3881de3c7605bc645024ac54672656ac77c104"} Oct 30 21:55:21.086498 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:21.086458 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" podStartSLOduration=2.086446446 podStartE2EDuration="2.086446446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:21.085064702 +0000 UTC m=+1583.280755822" watchObservedRunningTime="2026-10-30 21:55:21.086446446 +0000 UTC m=+1583.282137565" Oct 30 21:55:22.067941 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:22.067910 6808 generic.go:356] "Generic (PLEG): container finished" podID="3edf3409-61c0-4c08-8e7f-c179a40ccd59" containerID="08483574869c9db787450ec7c490ce096307a4582db12a6b20ae7682ad191690" exitCode=0 Oct 30 21:55:22.068329 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:22.067992 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" event={"ID":"3edf3409-61c0-4c08-8e7f-c179a40ccd59","Type":"ContainerDied","Data":"08483574869c9db787450ec7c490ce096307a4582db12a6b20ae7682ad191690"} Oct 30 21:55:23.204158 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.204135 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:23.266382 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.266352 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca\") pod \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " Oct 30 21:55:23.266528 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.266396 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca\") pod \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " Oct 30 21:55:23.266528 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.266456 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f\" (UniqueName: \"kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f\") pod \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\" (UID: \"3edf3409-61c0-4c08-8e7f-c179a40ccd59\") " Oct 30 21:55:23.266792 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.266764 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca" pod "3edf3409-61c0-4c08-8e7f-c179a40ccd59" (UID: "3edf3409-61c0-4c08-8e7f-c179a40ccd59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:23.268356 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.268332 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca" pod "3edf3409-61c0-4c08-8e7f-c179a40ccd59" (UID: "3edf3409-61c0-4c08-8e7f-c179a40ccd59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:23.268546 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.268527 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f" pod "3edf3409-61c0-4c08-8e7f-c179a40ccd59" (UID: "3edf3409-61c0-4c08-8e7f-c179a40ccd59"). InnerVolumeSpecName "kube-api-access-qwf4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:23.368131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.368071 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:23.368131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.368101 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edf3409-61c0-4c08-8e7f-c179a40ccd59-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:23.368131 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:23.368113 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwf4f\" (UniqueName: \"kubernetes.io/projected/3edf3409-61c0-4c08-8e7f-c179a40ccd59-kube-api-access-qwf4f\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.077864 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:24.077834 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" Oct 30 21:55:24.078052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:24.077871 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz" event={"ID":"3edf3409-61c0-4c08-8e7f-c179a40ccd59","Type":"ContainerDied","Data":"1af4c5b0b76d831ebcdd0a4e0a3881de3c7605bc645024ac54672656ac77c104"} Oct 30 21:55:24.078052 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:24.077924 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af4c5b0b76d831ebcdd0a4e0a3881de3c7605bc645024ac54672656ac77c104" Oct 30 21:55:35.734865 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:55:35.734827 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:55:39.043993 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.043961 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c"] Oct 30 21:55:39.044530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.044361 6808 cpu_manager.go:414] "containerMap: removing container" podUID="3edf3409-61c0-4c08-8e7f-c179a40ccd59" containerName="revoke-keys" Oct 30 21:55:39.044530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.044372 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edf3409-61c0-4c08-8e7f-c179a40ccd59" containerName="revoke-keys" Oct 30 21:55:39.044530 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.044444 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edf3409-61c0-4c08-8e7f-c179a40ccd59" containerName="revoke-keys" Oct 30 21:55:39.047911 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.047887 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.052456 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.052431 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:39.053722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.053698 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:55:39.053722 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.053719 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:39.055216 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.055181 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:39.057963 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.057926 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c"] Oct 30 21:55:39.139390 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.139356 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.139564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.139475 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.139564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.139532 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rnw\" (UniqueName: \"kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.240395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.240363 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.240548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.240414 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rnw\" (UniqueName: \"kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.240548 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.240476 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.241143 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.241118 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.242617 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.242591 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.252863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.252839 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rnw\" (UniqueName: \"kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw\") pod \"maas-api-revoke-sub-79805ca9-9456c\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.361611 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.361533 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:39.637441 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:39.637411 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c"] Oct 30 21:55:39.639958 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:55:39.639922 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b9319e_59c3_4b88_a2af_100cfa56f33a.slice/crio-220a06799eb0f1a641f4ba5e5d367548babb813ecd487a5a88ccbaa39e903bb6 WatchSource:0}: Error finding container 220a06799eb0f1a641f4ba5e5d367548babb813ecd487a5a88ccbaa39e903bb6: Status 404 returned error can't find the container with id 220a06799eb0f1a641f4ba5e5d367548babb813ecd487a5a88ccbaa39e903bb6 Oct 30 21:55:40.159157 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:40.159117 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" event={"ID":"60b9319e-59c3-4b88-a2af-100cfa56f33a","Type":"ContainerStarted","Data":"1c5bc9486d0745763ea3390a9b205e52c5d1103bb799cb39fb73685137f5fccf"} Oct 30 21:55:40.159590 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:40.159177 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" event={"ID":"60b9319e-59c3-4b88-a2af-100cfa56f33a","Type":"ContainerStarted","Data":"220a06799eb0f1a641f4ba5e5d367548babb813ecd487a5a88ccbaa39e903bb6"} Oct 30 21:55:40.179789 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:40.179746 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" podStartSLOduration=1.179734058 podStartE2EDuration="1.179734058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:40.17805209 +0000 UTC m=+1602.373743208" watchObservedRunningTime="2026-10-30 21:55:40.179734058 +0000 UTC m=+1602.375425180" Oct 30 21:55:41.165787 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:41.165744 6808 generic.go:356] "Generic (PLEG): container finished" podID="60b9319e-59c3-4b88-a2af-100cfa56f33a" containerID="1c5bc9486d0745763ea3390a9b205e52c5d1103bb799cb39fb73685137f5fccf" exitCode=0 Oct 30 21:55:41.166184 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:41.165827 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" event={"ID":"60b9319e-59c3-4b88-a2af-100cfa56f33a","Type":"ContainerDied","Data":"1c5bc9486d0745763ea3390a9b205e52c5d1103bb799cb39fb73685137f5fccf"} Oct 30 21:55:42.302686 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.302660 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:42.369776 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.369713 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca\") pod \"60b9319e-59c3-4b88-a2af-100cfa56f33a\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " Oct 30 21:55:42.370024 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.369999 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw\" (UniqueName: \"kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw\") pod \"60b9319e-59c3-4b88-a2af-100cfa56f33a\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " Oct 30 21:55:42.370164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.370051 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca\") pod \"60b9319e-59c3-4b88-a2af-100cfa56f33a\" (UID: \"60b9319e-59c3-4b88-a2af-100cfa56f33a\") " Oct 30 21:55:42.370164 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.370092 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca" pod "60b9319e-59c3-4b88-a2af-100cfa56f33a" (UID: "60b9319e-59c3-4b88-a2af-100cfa56f33a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:42.370501 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.370478 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:42.372137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.372111 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca" pod "60b9319e-59c3-4b88-a2af-100cfa56f33a" (UID: "60b9319e-59c3-4b88-a2af-100cfa56f33a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:42.372226 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.372179 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw" pod "60b9319e-59c3-4b88-a2af-100cfa56f33a" (UID: "60b9319e-59c3-4b88-a2af-100cfa56f33a"). InnerVolumeSpecName "kube-api-access-65rnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:42.471257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.471202 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rnw\" (UniqueName: \"kubernetes.io/projected/60b9319e-59c3-4b88-a2af-100cfa56f33a-kube-api-access-65rnw\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:42.471257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:42.471225 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9319e-59c3-4b88-a2af-100cfa56f33a-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:43.178613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:43.178586 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" Oct 30 21:55:43.178613 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:43.178598 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c" event={"ID":"60b9319e-59c3-4b88-a2af-100cfa56f33a","Type":"ContainerDied","Data":"220a06799eb0f1a641f4ba5e5d367548babb813ecd487a5a88ccbaa39e903bb6"} Oct 30 21:55:43.178835 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:43.178650 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220a06799eb0f1a641f4ba5e5d367548babb813ecd487a5a88ccbaa39e903bb6" Oct 30 21:55:56.938415 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.938383 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7"] Oct 30 21:55:56.938882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.938772 6808 cpu_manager.go:414] "containerMap: removing container" podUID="60b9319e-59c3-4b88-a2af-100cfa56f33a" containerName="revoke-keys" Oct 30 21:55:56.938882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.938785 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b9319e-59c3-4b88-a2af-100cfa56f33a" containerName="revoke-keys" Oct 30 21:55:56.938882 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.938848 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b9319e-59c3-4b88-a2af-100cfa56f33a" containerName="revoke-keys" Oct 30 21:55:56.941958 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.941930 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:56.949453 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.949434 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:56.949662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.949642 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:56.949744 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.949730 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:56.949823 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.949812 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:55:56.954562 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:56.954531 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7"] Oct 30 21:55:57.023398 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.023370 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.023558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.023403 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.023558 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.023504 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qx7\" (UniqueName: \"kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.124943 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.124915 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.125103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.124966 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.125103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.125081 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qx7\" (UniqueName: \"kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.125585 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.125557 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.127133 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.127116 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.137629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.137606 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qx7\" (UniqueName: \"kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7\") pod \"maas-api-revoke-sub-0b7e8edb-q62p7\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.255780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.255716 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:57.532163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:57.532135 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7"] Oct 30 21:55:57.533964 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:55:57.533927 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14963e8f_e4ef_4f2a_bd19_f04db6c3c71a.slice/crio-151c52fe8c1e51aa429ae945a40179e8afd17bba1497cd8bf9e9bc954ea4da31 WatchSource:0}: Error finding container 151c52fe8c1e51aa429ae945a40179e8afd17bba1497cd8bf9e9bc954ea4da31: Status 404 returned error can't find the container with id 151c52fe8c1e51aa429ae945a40179e8afd17bba1497cd8bf9e9bc954ea4da31 Oct 30 21:55:58.259885 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:58.259860 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" event={"ID":"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a","Type":"ContainerStarted","Data":"3be3dd5f603f29a5db2a99369d9e75f99af5d2563efb6e467e313e4428dc87f8"} Oct 30 21:55:58.260212 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:58.259911 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" event={"ID":"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a","Type":"ContainerStarted","Data":"151c52fe8c1e51aa429ae945a40179e8afd17bba1497cd8bf9e9bc954ea4da31"} Oct 30 21:55:59.265340 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.265307 6808 generic.go:356] "Generic (PLEG): container finished" podID="14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" containerID="3be3dd5f603f29a5db2a99369d9e75f99af5d2563efb6e467e313e4428dc87f8" exitCode=0 Oct 30 21:55:59.265754 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.265393 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" event={"ID":"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a","Type":"ContainerDied","Data":"3be3dd5f603f29a5db2a99369d9e75f99af5d2563efb6e467e313e4428dc87f8"} Oct 30 21:55:59.399680 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.399658 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:55:59.549916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.549841 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca\") pod \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " Oct 30 21:55:59.549916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.549880 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7\" (UniqueName: \"kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7\") pod \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " Oct 30 21:55:59.549916 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.549909 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca\") pod \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\" (UID: \"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a\") " Oct 30 21:55:59.550239 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.550218 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca" pod "14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" (UID: "14963e8f-e4ef-4f2a-bd19-f04db6c3c71a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:59.551995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.551970 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca" pod "14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" (UID: "14963e8f-e4ef-4f2a-bd19-f04db6c3c71a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:59.551995 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.551983 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7" pod "14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" (UID: "14963e8f-e4ef-4f2a-bd19-f04db6c3c71a"). InnerVolumeSpecName "kube-api-access-82qx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:59.651645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.651610 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:59.651645 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.651641 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qx7\" (UniqueName: \"kubernetes.io/projected/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-kube-api-access-82qx7\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:59.651828 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:55:59.651656 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:00.271087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:00.271011 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" Oct 30 21:56:00.271087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:00.271033 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7" event={"ID":"14963e8f-e4ef-4f2a-bd19-f04db6c3c71a","Type":"ContainerDied","Data":"151c52fe8c1e51aa429ae945a40179e8afd17bba1497cd8bf9e9bc954ea4da31"} Oct 30 21:56:00.271087 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:00.271082 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="151c52fe8c1e51aa429ae945a40179e8afd17bba1497cd8bf9e9bc954ea4da31" Oct 30 21:56:05.697249 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:56:05.697211 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:56:07.621438 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.621400 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt"] Oct 30 21:56:07.622000 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.621980 6808 cpu_manager.go:414] "containerMap: removing container" podUID="14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" containerName="revoke-keys" Oct 30 21:56:07.622103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.622001 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" containerName="revoke-keys" Oct 30 21:56:07.622163 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.622147 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" containerName="revoke-keys" Oct 30 21:56:07.625546 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.625523 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.633988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.633898 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:07.633988 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.633913 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:07.634145 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.634044 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:07.634349 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.634328 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:56:07.636561 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.636535 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt"] Oct 30 21:56:07.718519 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.718487 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.718673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.718529 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlhp\" (UniqueName: \"kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.718673 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.718650 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.819110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.819066 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlhp\" (UniqueName: \"kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.819307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.819184 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.819307 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.819255 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.819806 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.819775 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.821556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.821534 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.831874 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.831845 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlhp\" (UniqueName: \"kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp\") pod \"maas-api-revoke-sub-3d4156f1-c4lvt\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:07.940544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:07.940515 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:08.219812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:08.219724 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt"] Oct 30 21:56:08.225505 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:56:08.225472 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0ff57c_b97d_4b17_9c50_10e72767935a.slice/crio-56fc6e5cef34f7097b8f20791fe0f882057a37b8ef2501a633ff64cada951115 WatchSource:0}: Error finding container 56fc6e5cef34f7097b8f20791fe0f882057a37b8ef2501a633ff64cada951115: Status 404 returned error can't find the container with id 56fc6e5cef34f7097b8f20791fe0f882057a37b8ef2501a633ff64cada951115 Oct 30 21:56:08.328589 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:08.328545 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" event={"ID":"9d0ff57c-b97d-4b17-9c50-10e72767935a","Type":"ContainerStarted","Data":"56fc6e5cef34f7097b8f20791fe0f882057a37b8ef2501a633ff64cada951115"} Oct 30 21:56:09.335172 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:09.335081 6808 generic.go:356] "Generic (PLEG): container finished" podID="9d0ff57c-b97d-4b17-9c50-10e72767935a" containerID="cc4b56056b4a1c04b3fd104d303c52a3c9bb833a822a449df26ab7ee6aa68986" exitCode=0 Oct 30 21:56:09.335172 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:09.335131 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" event={"ID":"9d0ff57c-b97d-4b17-9c50-10e72767935a","Type":"ContainerDied","Data":"cc4b56056b4a1c04b3fd104d303c52a3c9bb833a822a449df26ab7ee6aa68986"} Oct 30 21:56:10.478780 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.478754 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:10.543203 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.543176 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca\") pod \"9d0ff57c-b97d-4b17-9c50-10e72767935a\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " Oct 30 21:56:10.543360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.543217 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca\") pod \"9d0ff57c-b97d-4b17-9c50-10e72767935a\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " Oct 30 21:56:10.543360 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.543242 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp\" (UniqueName: \"kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp\") pod \"9d0ff57c-b97d-4b17-9c50-10e72767935a\" (UID: \"9d0ff57c-b97d-4b17-9c50-10e72767935a\") " Oct 30 21:56:10.543621 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.543597 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca" pod "9d0ff57c-b97d-4b17-9c50-10e72767935a" (UID: "9d0ff57c-b97d-4b17-9c50-10e72767935a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:10.545395 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.545368 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca" pod "9d0ff57c-b97d-4b17-9c50-10e72767935a" (UID: "9d0ff57c-b97d-4b17-9c50-10e72767935a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:10.545516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.545404 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp" pod "9d0ff57c-b97d-4b17-9c50-10e72767935a" (UID: "9d0ff57c-b97d-4b17-9c50-10e72767935a"). InnerVolumeSpecName "kube-api-access-zjlhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.645103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.645067 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.645103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.645095 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0ff57c-b97d-4b17-9c50-10e72767935a-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.645103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:10.645106 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjlhp\" (UniqueName: \"kubernetes.io/projected/9d0ff57c-b97d-4b17-9c50-10e72767935a-kube-api-access-zjlhp\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.345985 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:11.345933 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" Oct 30 21:56:11.346137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:11.345983 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt" event={"ID":"9d0ff57c-b97d-4b17-9c50-10e72767935a","Type":"ContainerDied","Data":"56fc6e5cef34f7097b8f20791fe0f882057a37b8ef2501a633ff64cada951115"} Oct 30 21:56:11.346137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:11.346032 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56fc6e5cef34f7097b8f20791fe0f882057a37b8ef2501a633ff64cada951115" Oct 30 21:56:22.037593 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:22.037541 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h"] Oct 30 21:56:22.040229 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:22.039588 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fca89e-bwv8h"] Oct 30 21:56:23.929863 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:23.929815 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21be50d6-a832-443b-9e2f-f1e7f4f403b3" path="/var/lib/kubelet/pods/21be50d6-a832-443b-9e2f-f1e7f4f403b3/volumes" Oct 30 21:56:25.790642 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.790604 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt"] Oct 30 21:56:25.792463 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.792441 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c4lvt"] Oct 30 21:56:25.804406 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.804382 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj"] Oct 30 21:56:25.804742 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.804730 6808 cpu_manager.go:414] "containerMap: removing container" podUID="9d0ff57c-b97d-4b17-9c50-10e72767935a" containerName="revoke-keys" Oct 30 21:56:25.804795 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.804743 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0ff57c-b97d-4b17-9c50-10e72767935a" containerName="revoke-keys" Oct 30 21:56:25.804858 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.804838 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0ff57c-b97d-4b17-9c50-10e72767935a" containerName="revoke-keys" Oct 30 21:56:25.808081 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.808067 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.812332 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.812315 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:25.812416 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.812337 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:56:25.813230 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.813213 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:25.814623 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.814606 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:25.818418 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.818398 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj"] Oct 30 21:56:25.834195 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.834157 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.835216 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.835190 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.835313 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.835261 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85tr\" (UniqueName: \"kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.926738 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.926708 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0ff57c-b97d-4b17-9c50-10e72767935a" path="/var/lib/kubelet/pods/9d0ff57c-b97d-4b17-9c50-10e72767935a/volumes" Oct 30 21:56:25.936054 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.936036 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.936153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.936075 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85tr\" (UniqueName: \"kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.936153 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.936103 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.936759 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.936741 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.938261 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.938244 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:25.948611 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:25.948581 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85tr\" (UniqueName: \"kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr\") pod \"maas-api-revoke-sub-3d4156f1-wntjj\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:26.122886 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:26.122814 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:26.397662 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:26.397637 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj"] Oct 30 21:56:26.399661 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:56:26.399634 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f69f09a_167d_4241_83b6_176ed55203c1.slice/crio-d6f2162f8ae658492e66af93af14e90256faf51547c748020d551a0358067580 WatchSource:0}: Error finding container d6f2162f8ae658492e66af93af14e90256faf51547c748020d551a0358067580: Status 404 returned error can't find the container with id d6f2162f8ae658492e66af93af14e90256faf51547c748020d551a0358067580 Oct 30 21:56:26.439837 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:26.439809 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" event={"ID":"0f69f09a-167d-4241-83b6-176ed55203c1","Type":"ContainerStarted","Data":"d6f2162f8ae658492e66af93af14e90256faf51547c748020d551a0358067580"} Oct 30 21:56:27.445811 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:27.445778 6808 generic.go:356] "Generic (PLEG): container finished" podID="0f69f09a-167d-4241-83b6-176ed55203c1" containerID="ba3b0b571c5a86b6744882d20ee147d6b127f5b34b642db632ed2fe91bc7d467" exitCode=0 Oct 30 21:56:27.446189 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:27.445859 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" event={"ID":"0f69f09a-167d-4241-83b6-176ed55203c1","Type":"ContainerDied","Data":"ba3b0b571c5a86b6744882d20ee147d6b127f5b34b642db632ed2fe91bc7d467"} Oct 30 21:56:28.579175 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.579152 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:28.658609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.658581 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr\" (UniqueName: \"kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr\") pod \"0f69f09a-167d-4241-83b6-176ed55203c1\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " Oct 30 21:56:28.658757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.658623 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca\") pod \"0f69f09a-167d-4241-83b6-176ed55203c1\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " Oct 30 21:56:28.658757 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.658653 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca\") pod \"0f69f09a-167d-4241-83b6-176ed55203c1\" (UID: \"0f69f09a-167d-4241-83b6-176ed55203c1\") " Oct 30 21:56:28.659072 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.659042 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca" pod "0f69f09a-167d-4241-83b6-176ed55203c1" (UID: "0f69f09a-167d-4241-83b6-176ed55203c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:28.660665 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.660643 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca" pod "0f69f09a-167d-4241-83b6-176ed55203c1" (UID: "0f69f09a-167d-4241-83b6-176ed55203c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:28.660746 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.660696 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr" pod "0f69f09a-167d-4241-83b6-176ed55203c1" (UID: "0f69f09a-167d-4241-83b6-176ed55203c1"). InnerVolumeSpecName "kube-api-access-n85tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:28.760049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.759993 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85tr\" (UniqueName: \"kubernetes.io/projected/0f69f09a-167d-4241-83b6-176ed55203c1-kube-api-access-n85tr\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:28.760049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.760015 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:28.760049 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:28.760025 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69f09a-167d-4241-83b6-176ed55203c1-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.456495 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:29.456465 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" Oct 30 21:56:29.456672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:29.456508 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj" event={"ID":"0f69f09a-167d-4241-83b6-176ed55203c1","Type":"ContainerDied","Data":"d6f2162f8ae658492e66af93af14e90256faf51547c748020d551a0358067580"} Oct 30 21:56:29.456672 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:29.456555 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f2162f8ae658492e66af93af14e90256faf51547c748020d551a0358067580" Oct 30 21:56:32.027694 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:32.027662 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6"] Oct 30 21:56:32.030314 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:32.030289 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df361530-bdvz6"] Oct 30 21:56:33.926405 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:33.926373 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166c9225-d920-4f04-bd04-d043b1228bd0" path="/var/lib/kubelet/pods/166c9225-d920-4f04-bd04-d043b1228bd0/volumes" Oct 30 21:56:35.696902 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:56:35.696870 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:56:49.779125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:49.779072 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:56:49.779809 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:49.779772 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w8ptl" podUID="55d693b2-3c4e-48e2-977f-d4cbef44f3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:54.646826 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.646789 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg"] Oct 30 21:56:54.647322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.647211 6808 cpu_manager.go:414] "containerMap: removing container" podUID="0f69f09a-167d-4241-83b6-176ed55203c1" containerName="revoke-keys" Oct 30 21:56:54.647322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.647225 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f69f09a-167d-4241-83b6-176ed55203c1" containerName="revoke-keys" Oct 30 21:56:54.647322 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.647312 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f69f09a-167d-4241-83b6-176ed55203c1" containerName="revoke-keys" Oct 30 21:56:54.650385 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.650365 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.657311 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.657289 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:54.658643 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.658625 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:54.658718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.658661 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:54.658718 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.658700 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:56:54.665093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.665069 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg"] Oct 30 21:56:54.748359 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.748333 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.748493 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.748389 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.748544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.748521 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44t62\" (UniqueName: \"kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.849890 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.849853 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.850066 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.849927 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.850140 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.850066 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44t62\" (UniqueName: \"kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.850539 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.850507 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.852271 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.852248 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.863150 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.863132 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44t62\" (UniqueName: \"kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62\") pod \"maas-api-revoke-sub-e2fc9c5b-w79xg\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:54.963139 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:54.963087 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:55.241700 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:56:55.241658 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ccc4f4_8250_4597_9b56_251bb33d7924.slice/crio-9040c1f452c75a1aee5fba03f0a43283f4c72852a4f13cf7aee4dc1278ff23b3 WatchSource:0}: Error finding container 9040c1f452c75a1aee5fba03f0a43283f4c72852a4f13cf7aee4dc1278ff23b3: Status 404 returned error can't find the container with id 9040c1f452c75a1aee5fba03f0a43283f4c72852a4f13cf7aee4dc1278ff23b3 Oct 30 21:56:55.243103 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:55.243070 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg"] Oct 30 21:56:55.618199 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:55.618165 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" event={"ID":"74ccc4f4-8250-4597-9b56-251bb33d7924","Type":"ContainerStarted","Data":"b72c8295cc2729d184ea4dc3b1f5ac8260c850fd48958c5e4ca7234dcc26dbcc"} Oct 30 21:56:55.618346 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:55.618236 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" event={"ID":"74ccc4f4-8250-4597-9b56-251bb33d7924","Type":"ContainerStarted","Data":"9040c1f452c75a1aee5fba03f0a43283f4c72852a4f13cf7aee4dc1278ff23b3"} Oct 30 21:56:55.637556 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:55.637445 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" podStartSLOduration=1.637430251 podStartE2EDuration="1.637430251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:55.637166444 +0000 UTC m=+1677.832857563" watchObservedRunningTime="2026-10-30 21:56:55.637430251 +0000 UTC m=+1677.833121367" Oct 30 21:56:56.037851 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:56.037823 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56"] Oct 30 21:56:56.044564 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:56.044539 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e6911f-hch56"] Oct 30 21:56:56.624540 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:56.624504 6808 generic.go:356] "Generic (PLEG): container finished" podID="74ccc4f4-8250-4597-9b56-251bb33d7924" containerID="b72c8295cc2729d184ea4dc3b1f5ac8260c850fd48958c5e4ca7234dcc26dbcc" exitCode=0 Oct 30 21:56:56.624706 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:56.624585 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" event={"ID":"74ccc4f4-8250-4597-9b56-251bb33d7924","Type":"ContainerDied","Data":"b72c8295cc2729d184ea4dc3b1f5ac8260c850fd48958c5e4ca7234dcc26dbcc"} Oct 30 21:56:57.756843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.756820 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:57.774192 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.774171 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca\") pod \"74ccc4f4-8250-4597-9b56-251bb33d7924\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " Oct 30 21:56:57.774336 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.774243 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca\") pod \"74ccc4f4-8250-4597-9b56-251bb33d7924\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " Oct 30 21:56:57.774336 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.774278 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62\" (UniqueName: \"kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62\") pod \"74ccc4f4-8250-4597-9b56-251bb33d7924\" (UID: \"74ccc4f4-8250-4597-9b56-251bb33d7924\") " Oct 30 21:56:57.774527 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.774503 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca" pod "74ccc4f4-8250-4597-9b56-251bb33d7924" (UID: "74ccc4f4-8250-4597-9b56-251bb33d7924"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:57.774652 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.774635 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.776758 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.776734 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62" pod "74ccc4f4-8250-4597-9b56-251bb33d7924" (UID: "74ccc4f4-8250-4597-9b56-251bb33d7924"). InnerVolumeSpecName "kube-api-access-44t62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:57.776875 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.776849 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca" pod "74ccc4f4-8250-4597-9b56-251bb33d7924" (UID: "74ccc4f4-8250-4597-9b56-251bb33d7924"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:57.876093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.876060 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ccc4f4-8250-4597-9b56-251bb33d7924-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.876093 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.876095 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44t62\" (UniqueName: \"kubernetes.io/projected/74ccc4f4-8250-4597-9b56-251bb33d7924-kube-api-access-44t62\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.926370 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:57.926299 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd84627-40a3-472e-8d9c-b1aa22437cd4" path="/var/lib/kubelet/pods/7dd84627-40a3-472e-8d9c-b1aa22437cd4/volumes" Oct 30 21:56:58.636040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:58.636013 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" Oct 30 21:56:58.636040 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:58.636029 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg" event={"ID":"74ccc4f4-8250-4597-9b56-251bb33d7924","Type":"ContainerDied","Data":"9040c1f452c75a1aee5fba03f0a43283f4c72852a4f13cf7aee4dc1278ff23b3"} Oct 30 21:56:58.636248 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:56:58.636072 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9040c1f452c75a1aee5fba03f0a43283f4c72852a4f13cf7aee4dc1278ff23b3" Oct 30 21:57:04.878841 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.878807 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh"] Oct 30 21:57:04.879559 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.879206 6808 cpu_manager.go:414] "containerMap: removing container" podUID="74ccc4f4-8250-4597-9b56-251bb33d7924" containerName="revoke-keys" Oct 30 21:57:04.879559 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.879218 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ccc4f4-8250-4597-9b56-251bb33d7924" containerName="revoke-keys" Oct 30 21:57:04.879559 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.879285 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ccc4f4-8250-4597-9b56-251bb33d7924" containerName="revoke-keys" Oct 30 21:57:04.882703 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.882686 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:04.890928 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.890830 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:04.891256 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.891240 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:57:04.893932 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.893903 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh"] Oct 30 21:57:04.898309 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.898284 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:04.898445 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.898373 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:04.935797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.935766 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:04.935797 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.935824 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:04.936114 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:04.935897 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzd8\" (UniqueName: \"kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.037110 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.037080 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.037290 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.037136 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.037290 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.037154 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzd8\" (UniqueName: \"kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.037842 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.037816 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.039325 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.039304 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.050269 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.050245 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzd8\" (UniqueName: \"kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8\") pod \"maas-api-revoke-sub-5a29efb3-pr9fh\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.200287 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.200212 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:05.476544 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.476521 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh"] Oct 30 21:57:05.480993 ip-10-0-137-168 kubenswrapper[6808]: W1030 21:57:05.479214 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c20392_4e1c_41d6_809b_09c7b53e9838.slice/crio-e4bcbd1a58c52fde26ccc5d65fa043570c48405ff038896c1ccdcad3e1b720d9 WatchSource:0}: Error finding container e4bcbd1a58c52fde26ccc5d65fa043570c48405ff038896c1ccdcad3e1b720d9: Status 404 returned error can't find the container with id e4bcbd1a58c52fde26ccc5d65fa043570c48405ff038896c1ccdcad3e1b720d9 Oct 30 21:57:05.671295 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:05.671265 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" event={"ID":"00c20392-4e1c-41d6-809b-09c7b53e9838","Type":"ContainerStarted","Data":"e4bcbd1a58c52fde26ccc5d65fa043570c48405ff038896c1ccdcad3e1b720d9"} Oct 30 21:57:05.696743 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:57:05.696715 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:57:06.678516 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:06.678479 6808 generic.go:356] "Generic (PLEG): container finished" podID="00c20392-4e1c-41d6-809b-09c7b53e9838" containerID="2d231bf472e39d85af067ea90b5b04a4e85ac46dbcd17b25bb8c1236ef61fee5" exitCode=0 Oct 30 21:57:06.679077 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:06.678571 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" event={"ID":"00c20392-4e1c-41d6-809b-09c7b53e9838","Type":"ContainerDied","Data":"2d231bf472e39d85af067ea90b5b04a4e85ac46dbcd17b25bb8c1236ef61fee5"} Oct 30 21:57:08.140257 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.140229 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:08.268233 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.260002 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca\") pod \"00c20392-4e1c-41d6-809b-09c7b53e9838\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " Oct 30 21:57:08.292971 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.268514 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca\") pod \"00c20392-4e1c-41d6-809b-09c7b53e9838\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " Oct 30 21:57:08.294422 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.294396 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8\" (UniqueName: \"kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8\") pod \"00c20392-4e1c-41d6-809b-09c7b53e9838\" (UID: \"00c20392-4e1c-41d6-809b-09c7b53e9838\") " Oct 30 21:57:08.303028 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.268187 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca" pod "00c20392-4e1c-41d6-809b-09c7b53e9838" (UID: "00c20392-4e1c-41d6-809b-09c7b53e9838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:08.318417 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.318391 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca" pod "00c20392-4e1c-41d6-809b-09c7b53e9838" (UID: "00c20392-4e1c-41d6-809b-09c7b53e9838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:08.322215 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.322191 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8" pod "00c20392-4e1c-41d6-809b-09c7b53e9838" (UID: "00c20392-4e1c-41d6-809b-09c7b53e9838"). InnerVolumeSpecName "kube-api-access-2gzd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:08.396609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.396524 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-openshift-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.396609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.396565 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c20392-4e1c-41d6-809b-09c7b53e9838-maas-api-xks-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.396609 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.396582 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzd8\" (UniqueName: \"kubernetes.io/projected/00c20392-4e1c-41d6-809b-09c7b53e9838-kube-api-access-2gzd8\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.701459 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.701371 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" Oct 30 21:57:08.701629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.701374 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh" event={"ID":"00c20392-4e1c-41d6-809b-09c7b53e9838","Type":"ContainerDied","Data":"e4bcbd1a58c52fde26ccc5d65fa043570c48405ff038896c1ccdcad3e1b720d9"} Oct 30 21:57:08.701629 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:08.701498 6808 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4bcbd1a58c52fde26ccc5d65fa043570c48405ff038896c1ccdcad3e1b720d9" Oct 30 21:57:35.697021 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:57:35.696987 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:57:45.028202 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:45.028119 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr"] Oct 30 21:57:45.031526 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:45.031497 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fq9jr"] Oct 30 21:57:45.925792 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:57:45.925752 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d441e2-4d81-4975-a2db-5ed9aba58e58" path="/var/lib/kubelet/pods/84d441e2-4d81-4975-a2db-5ed9aba58e58/volumes" Oct 30 21:58:04.948242 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:04.948209 6808 scope.go:122] "RemoveContainer" containerID="1bbe6ab926b6b6b666a5837aad77c082d56983e197807146b45ef561477be6e4" Oct 30 21:58:04.956887 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:04.956867 6808 scope.go:122] "RemoveContainer" containerID="c3c66a4ca5a12ed8bc4450ce98d959649f61019de89851debcd1b0542dc0e69a" Oct 30 21:58:04.964549 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:04.964532 6808 scope.go:122] "RemoveContainer" containerID="dc27e4baf255c4b2579ec5cee2ab22368cca3d79c18c8f699a6b0173ea854059" Oct 30 21:58:05.697515 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:58:05.697482 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:58:35.697447 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:58:35.697406 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:58:37.026843 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:37.026811 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw"] Oct 30 21:58:37.031332 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:37.031308 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jjncw"] Oct 30 21:58:37.926658 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:37.926627 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ff3943-5068-4e2b-b26a-f455d06f6c5a" path="/var/lib/kubelet/pods/76ff3943-5068-4e2b-b26a-f455d06f6c5a/volumes" Oct 30 21:58:56.035125 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:56.035093 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd"] Oct 30 21:58:56.038029 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:56.038004 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bcpzd"] Oct 30 21:58:57.926302 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:58:57.926262 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a8452c-ac08-4ae1-8a6e-097907b016d8" path="/var/lib/kubelet/pods/e8a8452c-ac08-4ae1-8a6e-097907b016d8/volumes" Oct 30 21:59:04.995083 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:04.995048 6808 scope.go:122] "RemoveContainer" containerID="60d0c9d0168b90a90a72c021ef952fb8dea4b7a0107a9f74d23932a8900568d4" Oct 30 21:59:05.697294 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:59:05.697260 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:59:07.023531 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:07.023491 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4"] Oct 30 21:59:07.026409 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:07.026382 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hhgn4"] Oct 30 21:59:07.926962 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:07.926909 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3817c72e-120c-47e6-8853-7b3166117104" path="/var/lib/kubelet/pods/3817c72e-120c-47e6-8853-7b3166117104/volumes" Oct 30 21:59:33.025244 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:33.025213 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92"] Oct 30 21:59:33.027812 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:33.027786 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-98r92"] Oct 30 21:59:33.927025 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:33.926983 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426acc21-44a4-47c1-93dc-786b0ea894bd" path="/var/lib/kubelet/pods/426acc21-44a4-47c1-93dc-786b0ea894bd/volumes" Oct 30 21:59:35.696675 ip-10-0-137-168 kubenswrapper[6808]: E1030 21:59:35.696640 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 21:59:44.050146 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:44.050109 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt"] Oct 30 21:59:44.052762 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:44.052737 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xz2rt"] Oct 30 21:59:45.927137 ip-10-0-137-168 kubenswrapper[6808]: I1030 21:59:45.927102 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bfd8f6-29a6-4ab6-827e-11ce4472d85a" path="/var/lib/kubelet/pods/e9bfd8f6-29a6-4ab6-827e-11ce4472d85a/volumes" Oct 30 22:00:05.016170 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:05.016136 6808 scope.go:122] "RemoveContainer" containerID="e1d425f02001224649e38901f9423b00318db34b7b056d53e07061009230ff39" Oct 30 22:00:05.024814 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:05.024784 6808 scope.go:122] "RemoveContainer" containerID="714e87487774ec343281f0281255b2273677d4e37fa9488f7c21b7ba9c4e57cd" Oct 30 22:00:05.032667 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:05.032646 6808 scope.go:122] "RemoveContainer" containerID="0a193cd78ae9c88677be5aa4f632d202afe3039afbe4538a9d864ff5429823ed" Oct 30 22:00:05.697653 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:00:05.697614 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:00:06.030123 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:06.030035 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl"] Oct 30 22:00:06.034649 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:06.034612 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-96dsl"] Oct 30 22:00:07.928149 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:07.928114 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8137d0a1-3b34-42f2-97bd-77b8e24b26bb" path="/var/lib/kubelet/pods/8137d0a1-3b34-42f2-97bd-77b8e24b26bb/volumes" Oct 30 22:00:24.027979 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:24.027932 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz"] Oct 30 22:00:24.032109 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:24.032084 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7rzgz"] Oct 30 22:00:25.925861 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:25.925823 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edf3409-61c0-4c08-8e7f-c179a40ccd59" path="/var/lib/kubelet/pods/3edf3409-61c0-4c08-8e7f-c179a40ccd59/volumes" Oct 30 22:00:35.698244 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:00:35.698203 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:00:43.026492 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:43.026305 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c"] Oct 30 22:00:43.028272 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:43.028248 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9456c"] Oct 30 22:00:43.926967 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:00:43.926916 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b9319e-59c3-4b88-a2af-100cfa56f33a" path="/var/lib/kubelet/pods/60b9319e-59c3-4b88-a2af-100cfa56f33a/volumes" Oct 30 22:01:00.067805 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:00.067772 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7"] Oct 30 22:01:00.068968 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:00.068926 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q62p7"] Oct 30 22:01:01.928809 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:01.928767 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14963e8f-e4ef-4f2a-bd19-f04db6c3c71a" path="/var/lib/kubelet/pods/14963e8f-e4ef-4f2a-bd19-f04db6c3c71a/volumes" Oct 30 22:01:05.060609 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:05.060578 6808 scope.go:122] "RemoveContainer" containerID="c43b69e6023c513e4f825913a46063bf47eeb21ca14a99f80d999c70638662a7" Oct 30 22:01:05.069075 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:05.069056 6808 scope.go:122] "RemoveContainer" containerID="d53c60f83b298dee8991b7d54e8d260c13d8e66dedbc216ef77efe35fb9c6ec3" Oct 30 22:01:05.076581 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:05.076564 6808 scope.go:122] "RemoveContainer" containerID="0c11b8527c9ae36eaece8b3c4c1af39847e3e7e272c0cf5c02d43901e759acb8" Oct 30 22:01:05.697012 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:01:05.696977 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:01:29.026723 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:29.026683 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj"] Oct 30 22:01:29.034088 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:29.034061 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wntjj"] Oct 30 22:01:29.926903 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:29.926870 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f69f09a-167d-4241-83b6-176ed55203c1" path="/var/lib/kubelet/pods/0f69f09a-167d-4241-83b6-176ed55203c1/volumes" Oct 30 22:01:35.698108 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:01:35.698068 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:01:58.024900 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:58.024863 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg"] Oct 30 22:01:58.028246 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:58.028220 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w79xg"] Oct 30 22:01:59.926795 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:01:59.926761 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ccc4f4-8250-4597-9b56-251bb33d7924" path="/var/lib/kubelet/pods/74ccc4f4-8250-4597-9b56-251bb33d7924/volumes" Oct 30 22:02:05.105766 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:02:05.105732 6808 scope.go:122] "RemoveContainer" containerID="08483574869c9db787450ec7c490ce096307a4582db12a6b20ae7682ad191690" Oct 30 22:02:05.114158 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:02:05.114142 6808 scope.go:122] "RemoveContainer" containerID="3be3dd5f603f29a5db2a99369d9e75f99af5d2563efb6e467e313e4428dc87f8" Oct 30 22:02:05.121445 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:02:05.121428 6808 scope.go:122] "RemoveContainer" containerID="1c5bc9486d0745763ea3390a9b205e52c5d1103bb799cb39fb73685137f5fccf" Oct 30 22:02:05.697324 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:02:05.697288 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:02:09.027790 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:02:09.027751 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh"] Oct 30 22:02:09.030604 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:02:09.030578 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pr9fh"] Oct 30 22:02:09.926792 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:02:09.926753 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c20392-4e1c-41d6-809b-09c7b53e9838" path="/var/lib/kubelet/pods/00c20392-4e1c-41d6-809b-09c7b53e9838/volumes" Oct 30 22:02:35.698124 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:02:35.698081 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:03:05.145943 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:03:05.145911 6808 scope.go:122] "RemoveContainer" containerID="ba3b0b571c5a86b6744882d20ee147d6b127f5b34b642db632ed2fe91bc7d467" Oct 30 22:03:05.155446 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:03:05.155425 6808 scope.go:122] "RemoveContainer" containerID="cc4b56056b4a1c04b3fd104d303c52a3c9bb833a822a449df26ab7ee6aa68986" Oct 30 22:03:05.163880 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:03:05.163861 6808 scope.go:122] "RemoveContainer" containerID="b72c8295cc2729d184ea4dc3b1f5ac8260c850fd48958c5e4ca7234dcc26dbcc" Oct 30 22:03:05.698258 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:03:05.698156 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:03:35.696978 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:03:35.696923 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:04:05.190742 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:05.190670 6808 scope.go:122] "RemoveContainer" containerID="2d231bf472e39d85af067ea90b5b04a4e85ac46dbcd17b25bb8c1236ef61fee5" Oct 30 22:04:05.697748 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:04:05.697710 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:04:16.670098 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:16.670043 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.670745 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:16.670691 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.760100 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:16.760063 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.760421 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:16.760374 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:25.406635 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.406605 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m"] Oct 30 22:04:25.407118 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.406798 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" podUID="f264a47c-660d-4427-a489-7218add05706" containerName="manager" containerID="cri-o://f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99" gracePeriod=10 Oct 30 22:04:25.666333 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.666251 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 22:04:25.702230 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.702197 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc\" (UniqueName: \"kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc\") pod \"f264a47c-660d-4427-a489-7218add05706\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " Oct 30 22:04:25.702423 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.702404 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume\") pod \"f264a47c-660d-4427-a489-7218add05706\" (UID: \"f264a47c-660d-4427-a489-7218add05706\") " Oct 30 22:04:25.702989 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.702937 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume" pod "f264a47c-660d-4427-a489-7218add05706" (UID: "f264a47c-660d-4427-a489-7218add05706"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:25.705207 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.705176 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc" pod "f264a47c-660d-4427-a489-7218add05706" (UID: "f264a47c-660d-4427-a489-7218add05706"). InnerVolumeSpecName "kube-api-access-9c2dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:25.806989 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.805007 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2dc\" (UniqueName: \"kubernetes.io/projected/f264a47c-660d-4427-a489-7218add05706-kube-api-access-9c2dc\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.806989 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.805045 6808 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f264a47c-660d-4427-a489-7218add05706-extensions-socket-volume\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.818075 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.818036 6808 generic.go:356] "Generic (PLEG): container finished" podID="f264a47c-660d-4427-a489-7218add05706" containerID="f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99" exitCode=0 Oct 30 22:04:25.820797 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.820757 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" Oct 30 22:04:25.822255 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.822020 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" event={"ID":"f264a47c-660d-4427-a489-7218add05706","Type":"ContainerDied","Data":"f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99"} Oct 30 22:04:25.822255 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.822102 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m" event={"ID":"f264a47c-660d-4427-a489-7218add05706","Type":"ContainerDied","Data":"848da59420d3d89b7670af71ee6a3a165b56137f50f563254dc3b811737f698d"} Oct 30 22:04:25.822255 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.822134 6808 scope.go:122] "RemoveContainer" containerID="f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99" Oct 30 22:04:25.842013 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.841984 6808 scope.go:122] "RemoveContainer" containerID="f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99" Oct 30 22:04:25.842404 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:04:25.842384 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99\": container with ID starting with f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99 not found: ID does not exist" containerID="f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99" Oct 30 22:04:25.842537 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.842413 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99"} err="failed to get container status \"f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99\": rpc error: code = NotFound desc = could not find container \"f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99\": container with ID starting with f3e452fbac565c7fbe8e8a4a7be84fe105b8cce05ba3a901c86cb30292861d99 not found: ID does not exist" Oct 30 22:04:25.854592 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.854565 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m"] Oct 30 22:04:25.859852 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.859826 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8s64m"] Oct 30 22:04:25.926499 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:04:25.926444 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f264a47c-660d-4427-a489-7218add05706" path="/var/lib/kubelet/pods/f264a47c-660d-4427-a489-7218add05706/volumes" Oct 30 22:04:35.698233 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:04:35.698192 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:05:05.697230 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:05:05.697186 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:05:35.697439 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:05:35.697404 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:05:46.672973 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:05:46.672071 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:46.672973 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:05:46.672184 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hxfjm" podUID="ce40587d-41ac-4215-9c6c-f0298ebec6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:46.751119 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:05:46.751074 6808 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wtthn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:46.752353 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:05:46.752314 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wtthn" podUID="c3699ff6-469e-42ec-9f4d-c9e124f1f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:05.697228 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:06:05.697185 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:06:24.436601 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.436568 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t"] Oct 30 22:06:24.437146 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.436905 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f264a47c-660d-4427-a489-7218add05706" containerName="manager" Oct 30 22:06:24.437146 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.436915 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f264a47c-660d-4427-a489-7218add05706" containerName="manager" Oct 30 22:06:24.437146 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.436943 6808 cpu_manager.go:414] "containerMap: removing container" podUID="00c20392-4e1c-41d6-809b-09c7b53e9838" containerName="revoke-keys" Oct 30 22:06:24.437146 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.436961 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c20392-4e1c-41d6-809b-09c7b53e9838" containerName="revoke-keys" Oct 30 22:06:24.437146 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.437028 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f264a47c-660d-4427-a489-7218add05706" containerName="manager" Oct 30 22:06:24.437146 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.437042 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c20392-4e1c-41d6-809b-09c7b53e9838" containerName="revoke-keys" Oct 30 22:06:24.447730 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.447702 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.453913 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.453888 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:24.454634 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.454215 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:24.455623 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.455601 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t"] Oct 30 22:06:24.466380 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.466176 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:06:24.466380 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.466255 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4ttw5\"" Oct 30 22:06:24.466380 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.466352 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0a9006-serving-cert\"" Oct 30 22:06:24.501315 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.501281 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969qb\" (UniqueName: \"kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.501474 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.501346 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.501474 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.501397 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.501474 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.501443 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.603422 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.603385 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.603613 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.603459 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.603613 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.603514 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.603727 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.603650 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969qb\" (UniqueName: \"kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.604288 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.604261 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.605962 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.605922 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.606134 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.606107 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.618523 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.618496 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-969qb\" (UniqueName: \"kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb\") pod \"maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:24.763097 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:24.763012 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:25.039328 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:25.039146 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t"] Oct 30 22:06:25.042140 ip-10-0-137-168 kubenswrapper[6808]: W1030 22:06:25.042111 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aec047_9205_4fe5_bfbd_0e0dad43ff9e.slice/crio-1138a4717932ac76fddfa26f843e11c12e10dbc08a835ecec4fabc2107adab4d WatchSource:0}: Error finding container 1138a4717932ac76fddfa26f843e11c12e10dbc08a835ecec4fabc2107adab4d: Status 404 returned error can't find the container with id 1138a4717932ac76fddfa26f843e11c12e10dbc08a835ecec4fabc2107adab4d Oct 30 22:06:25.043475 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:25.043458 6808 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:06:25.330828 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:25.330735 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" event={"ID":"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e","Type":"ContainerStarted","Data":"1138a4717932ac76fddfa26f843e11c12e10dbc08a835ecec4fabc2107adab4d"} Oct 30 22:06:26.336257 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:26.336219 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" event={"ID":"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e","Type":"ContainerStarted","Data":"d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502"} Oct 30 22:06:26.336644 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:26.336339 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:26.357544 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:26.357506 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" podStartSLOduration=1.8815437130000001 podStartE2EDuration="2.357494012s" totalImagesPullingTime="475.950299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.356403332 +0000 UTC m=+2248.552094445" watchObservedRunningTime="2026-10-30 22:06:26.357494012 +0000 UTC m=+2248.553185148" Oct 30 22:06:32.347617 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:32.347584 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:06:35.698037 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:06:35.698000 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:06:40.157482 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.157410 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj"] Oct 30 22:06:40.160893 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.160866 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.191796 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.191756 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0a9006\"/\"kube-root-ca.crt\"" Oct 30 22:06:40.191796 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.191770 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0a9006\"/\"openshift-service-ca.crt\"" Oct 30 22:06:40.191989 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.191801 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0a9006\"/\"default-dockercfg-r2gvd\"" Oct 30 22:06:40.193223 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.193179 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0a9006\"/\"simulator-single-main-0a9006-kserve-self-signed-certs\"" Oct 30 22:06:40.195749 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.194918 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj"] Oct 30 22:06:40.267535 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.267504 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.267696 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.267549 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.267696 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.267612 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.267772 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.267700 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7d4\" (UniqueName: \"kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.267772 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.267740 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.267861 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.267839 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369076 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369047 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369223 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369083 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369223 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369108 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369223 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369141 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7d4\" (UniqueName: \"kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369223 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369165 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369223 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369207 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369522 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369500 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369588 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369544 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.369588 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.369574 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.371562 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.371542 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.371794 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.371777 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.388515 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.388450 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7d4\" (UniqueName: \"kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4\") pod \"simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.474391 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.474313 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:40.768421 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:40.768305 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj"] Oct 30 22:06:40.771240 ip-10-0-137-168 kubenswrapper[6808]: W1030 22:06:40.771200 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda266b30a_e59e_4a2e_b59f_6ec17b21e5b3.slice/crio-425424dabfd2d27cdcac20ad851d2e01a2301f040990c4c0237fd4af16c3f84a WatchSource:0}: Error finding container 425424dabfd2d27cdcac20ad851d2e01a2301f040990c4c0237fd4af16c3f84a: Status 404 returned error can't find the container with id 425424dabfd2d27cdcac20ad851d2e01a2301f040990c4c0237fd4af16c3f84a Oct 30 22:06:41.420133 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:41.420099 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" event={"ID":"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3","Type":"ContainerStarted","Data":"cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e"} Oct 30 22:06:41.420133 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:41.420137 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" event={"ID":"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3","Type":"ContainerStarted","Data":"425424dabfd2d27cdcac20ad851d2e01a2301f040990c4c0237fd4af16c3f84a"} Oct 30 22:06:41.443519 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:41.443474 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" podStartSLOduration=1.443456742 podStartE2EDuration="1.443456742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:41.443034951 +0000 UTC m=+2263.638726072" watchObservedRunningTime="2026-10-30 22:06:41.443456742 +0000 UTC m=+2263.639147862" Oct 30 22:06:41.475366 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:41.475341 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:50.475012 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:50.474980 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:50.487602 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:50.487577 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:06:51.483037 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:06:51.483008 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:07:05.697599 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:07:05.697559 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:07:26.984695 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:26.984654 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj"] Oct 30 22:07:32.297308 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:32.297270 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t"] Oct 30 22:07:32.297849 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:32.297498 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" podUID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" containerName="maas-api" containerID="cri-o://d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502" gracePeriod=30 Oct 30 22:07:32.341313 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:32.341264 6808 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" podUID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 22:07:33.540268 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.540239 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:07:33.615211 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.615180 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls\") pod \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " Oct 30 22:07:33.615381 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.615217 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb\" (UniqueName: \"kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb\") pod \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " Oct 30 22:07:33.615381 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.615251 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs\") pod \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " Oct 30 22:07:33.615513 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.615390 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca\") pod \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\" (UID: \"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e\") " Oct 30 22:07:33.615779 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.615750 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca" pod "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" (UID: "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:33.617380 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.617360 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs" pod "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" (UID: "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:33.617483 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.617430 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb" pod "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" (UID: "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e"). InnerVolumeSpecName "kube-api-access-969qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:33.617483 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.617460 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls" pod "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" (UID: "f8aec047-9205-4fe5-bfbd-0e0dad43ff9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:33.670401 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.670289 6808 generic.go:356] "Generic (PLEG): container finished" podID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" containerID="d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502" exitCode=0 Oct 30 22:07:33.670401 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.670347 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" Oct 30 22:07:33.672539 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.670399 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" event={"ID":"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e","Type":"ContainerDied","Data":"d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502"} Oct 30 22:07:33.672539 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.670460 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t" event={"ID":"f8aec047-9205-4fe5-bfbd-0e0dad43ff9e","Type":"ContainerDied","Data":"1138a4717932ac76fddfa26f843e11c12e10dbc08a835ecec4fabc2107adab4d"} Oct 30 22:07:33.672539 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.670941 6808 scope.go:122] "RemoveContainer" containerID="d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502" Oct 30 22:07:33.680843 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.680822 6808 scope.go:122] "RemoveContainer" containerID="d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502" Oct 30 22:07:33.681115 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:07:33.681097 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502\": container with ID starting with d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502 not found: ID does not exist" containerID="d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502" Oct 30 22:07:33.681175 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.681124 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502"} err="failed to get container status \"d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502\": rpc error: code = NotFound desc = could not find container \"d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502\": container with ID starting with d72a58599ef89e0e42c13ea44f8b052110d020ef28783b6ee06970892c683502 not found: ID does not exist" Oct 30 22:07:33.694095 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.694075 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t"] Oct 30 22:07:33.698055 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.698035 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a9006-66bf768778-66h4t"] Oct 30 22:07:33.716414 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.716390 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:33.716414 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.716410 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969qb\" (UniqueName: \"kubernetes.io/projected/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-kube-api-access-969qb\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:33.716540 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.716426 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:33.716540 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.716439 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:33.927253 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:33.927225 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" path="/var/lib/kubelet/pods/f8aec047-9205-4fe5-bfbd-0e0dad43ff9e/volumes" Oct 30 22:07:35.698412 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:07:35.698375 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:07:42.004648 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.004590 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" podUID="a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" containerName="main" containerID="cri-o://cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e" gracePeriod=45 Oct 30 22:07:42.247674 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.247652 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:07:42.396869 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.396838 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm\") pod \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " Oct 30 22:07:42.397055 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.396892 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache\") pod \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " Oct 30 22:07:42.397055 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.396992 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4\" (UniqueName: \"kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4\") pod \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " Oct 30 22:07:42.397157 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397066 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs\") pod \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " Oct 30 22:07:42.397157 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397140 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache" pod "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" (UID: "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:42.397266 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397188 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home\") pod \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " Oct 30 22:07:42.397266 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397231 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir\") pod \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\" (UID: \"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3\") " Oct 30 22:07:42.397369 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397297 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home" pod "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" (UID: "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:42.397419 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397388 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir" pod "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" (UID: "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:42.397577 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397561 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.397644 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397583 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.397644 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.397597 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.399176 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.399157 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm" pod "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" (UID: "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:42.399555 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.399534 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs" pod "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" (UID: "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:42.399632 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.399570 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4" pod "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" (UID: "a266b30a-e59e-4a2e-b59f-6ec17b21e5b3"). InnerVolumeSpecName "kube-api-access-rm7d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:42.498695 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.498668 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.498695 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.498693 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.498850 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.498708 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm7d4\" (UniqueName: \"kubernetes.io/projected/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3-kube-api-access-rm7d4\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.711783 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.711706 6808 generic.go:356] "Generic (PLEG): container finished" podID="a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" containerID="cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e" exitCode=0 Oct 30 22:07:42.711783 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.711746 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" event={"ID":"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3","Type":"ContainerDied","Data":"cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e"} Oct 30 22:07:42.711783 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.711747 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" Oct 30 22:07:42.711783 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.711781 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj" event={"ID":"a266b30a-e59e-4a2e-b59f-6ec17b21e5b3","Type":"ContainerDied","Data":"425424dabfd2d27cdcac20ad851d2e01a2301f040990c4c0237fd4af16c3f84a"} Oct 30 22:07:42.712101 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.711800 6808 scope.go:122] "RemoveContainer" containerID="cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e" Oct 30 22:07:42.720714 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.720696 6808 scope.go:122] "RemoveContainer" containerID="cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e" Oct 30 22:07:42.720997 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:07:42.720973 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e\": container with ID starting with cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e not found: ID does not exist" containerID="cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e" Oct 30 22:07:42.721071 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.721011 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e"} err="failed to get container status \"cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e\": rpc error: code = NotFound desc = could not find container \"cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e\": container with ID starting with cb7e1732aa9693729e24f8a0060819db139e2b63a0d30d0c5b228ded7e955f6e not found: ID does not exist" Oct 30 22:07:42.733866 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.733843 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj"] Oct 30 22:07:42.737800 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:42.737781 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0a9006/simulator-single-main-0a9006-kserve-558bc5cd58-mhjqj"] Oct 30 22:07:43.927271 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:43.927229 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" path="/var/lib/kubelet/pods/a266b30a-e59e-4a2e-b59f-6ec17b21e5b3/volumes" Oct 30 22:07:44.147501 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.147466 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq"] Oct 30 22:07:44.147883 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.147866 6808 cpu_manager.go:414] "containerMap: removing container" podUID="a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" containerName="main" Oct 30 22:07:44.148001 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.147886 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" containerName="main" Oct 30 22:07:44.148001 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.147940 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" containerName="maas-api" Oct 30 22:07:44.148001 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.147964 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" containerName="maas-api" Oct 30 22:07:44.148170 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.148072 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aec047-9205-4fe5-bfbd-0e0dad43ff9e" containerName="maas-api" Oct 30 22:07:44.148170 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.148089 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="a266b30a-e59e-4a2e-b59f-6ec17b21e5b3" containerName="main" Oct 30 22:07:44.153496 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.153475 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.165328 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.165302 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a86449bf-kserve-self-signed-certs\"" Oct 30 22:07:44.165498 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.165373 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 22:07:44.165498 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.165410 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 22:07:44.165652 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.165534 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k8tnn\"" Oct 30 22:07:44.166667 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.166643 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq"] Oct 30 22:07:44.213882 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.213805 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.213882 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.213865 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z676g\" (UniqueName: \"kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.214059 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.214024 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.214157 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.214144 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.214196 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.214165 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.214278 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.214257 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.314791 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.314751 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z676g\" (UniqueName: \"kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315026 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.314803 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315026 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.314887 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315026 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.314910 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315026 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.314979 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315026 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.315014 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315326 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.315298 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315389 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.315338 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.315478 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.315459 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.317167 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.317151 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.317618 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.317596 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.336584 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.336556 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z676g\" (UniqueName: \"kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g\") pod \"e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.468552 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.468471 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:44.743661 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:44.743635 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq"] Oct 30 22:07:44.746187 ip-10-0-137-168 kubenswrapper[6808]: W1030 22:07:44.746153 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99191bf6_7c89_4200_ba11_1dcbcee37c3d.slice/crio-a799b46c5d55edc2b538195a7aa2a556582abbee0c79ebf49978bf0ad2c47a6b WatchSource:0}: Error finding container a799b46c5d55edc2b538195a7aa2a556582abbee0c79ebf49978bf0ad2c47a6b: Status 404 returned error can't find the container with id a799b46c5d55edc2b538195a7aa2a556582abbee0c79ebf49978bf0ad2c47a6b Oct 30 22:07:45.727873 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:45.727841 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" event={"ID":"99191bf6-7c89-4200-ba11-1dcbcee37c3d","Type":"ContainerStarted","Data":"b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0"} Oct 30 22:07:45.728281 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:45.727880 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" event={"ID":"99191bf6-7c89-4200-ba11-1dcbcee37c3d","Type":"ContainerStarted","Data":"a799b46c5d55edc2b538195a7aa2a556582abbee0c79ebf49978bf0ad2c47a6b"} Oct 30 22:07:45.749816 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:45.749765 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" podStartSLOduration=1.74974948 podStartE2EDuration="1.74974948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:45.748489699 +0000 UTC m=+2327.944180829" watchObservedRunningTime="2026-10-30 22:07:45.74974948 +0000 UTC m=+2327.945440594" Oct 30 22:07:46.468973 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:46.468623 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:54.468590 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:54.468557 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:54.480682 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:54.480657 6808 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:07:54.784932 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:07:54.784859 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:08:05.735766 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:08:05.735728 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:08:29.202586 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:29.202546 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq"] Oct 30 22:08:35.730000 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:08:35.729935 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:08:44.223353 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.223291 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" podUID="99191bf6-7c89-4200-ba11-1dcbcee37c3d" containerName="main" containerID="cri-o://b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0" gracePeriod=45 Oct 30 22:08:44.465990 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.465967 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:08:44.535029 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.534941 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir\") pod \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " Oct 30 22:08:44.535180 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535024 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache\") pod \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " Oct 30 22:08:44.535180 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535088 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs\") pod \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " Oct 30 22:08:44.535180 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535158 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home\") pod \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " Oct 30 22:08:44.535348 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535212 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g\" (UniqueName: \"kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g\") pod \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " Oct 30 22:08:44.535348 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535213 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir" pod "99191bf6-7c89-4200-ba11-1dcbcee37c3d" (UID: "99191bf6-7c89-4200-ba11-1dcbcee37c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:44.535348 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535272 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm\") pod \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\" (UID: \"99191bf6-7c89-4200-ba11-1dcbcee37c3d\") " Oct 30 22:08:44.535348 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535276 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache" pod "99191bf6-7c89-4200-ba11-1dcbcee37c3d" (UID: "99191bf6-7c89-4200-ba11-1dcbcee37c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:44.535523 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535407 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home" pod "99191bf6-7c89-4200-ba11-1dcbcee37c3d" (UID: "99191bf6-7c89-4200-ba11-1dcbcee37c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:44.535636 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535617 6808 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-home\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:44.535758 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535637 6808 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tmp-dir\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:44.535758 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.535659 6808 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-model-cache\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:44.537398 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.537371 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs" pod "99191bf6-7c89-4200-ba11-1dcbcee37c3d" (UID: "99191bf6-7c89-4200-ba11-1dcbcee37c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:44.537398 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.537386 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm" pod "99191bf6-7c89-4200-ba11-1dcbcee37c3d" (UID: "99191bf6-7c89-4200-ba11-1dcbcee37c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:44.537532 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.537413 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g" pod "99191bf6-7c89-4200-ba11-1dcbcee37c3d" (UID: "99191bf6-7c89-4200-ba11-1dcbcee37c3d"). InnerVolumeSpecName "kube-api-access-z676g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:44.636453 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.636430 6808 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99191bf6-7c89-4200-ba11-1dcbcee37c3d-tls-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:44.636453 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.636451 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z676g\" (UniqueName: \"kubernetes.io/projected/99191bf6-7c89-4200-ba11-1dcbcee37c3d-kube-api-access-z676g\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:44.636595 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:44.636460 6808 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99191bf6-7c89-4200-ba11-1dcbcee37c3d-dshm\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:45.055501 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.055465 6808 generic.go:356] "Generic (PLEG): container finished" podID="99191bf6-7c89-4200-ba11-1dcbcee37c3d" containerID="b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0" exitCode=0 Oct 30 22:08:45.055673 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.055514 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" Oct 30 22:08:45.055673 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.055549 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" event={"ID":"99191bf6-7c89-4200-ba11-1dcbcee37c3d","Type":"ContainerDied","Data":"b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0"} Oct 30 22:08:45.055673 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.055592 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq" event={"ID":"99191bf6-7c89-4200-ba11-1dcbcee37c3d","Type":"ContainerDied","Data":"a799b46c5d55edc2b538195a7aa2a556582abbee0c79ebf49978bf0ad2c47a6b"} Oct 30 22:08:45.055673 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.055609 6808 scope.go:122] "RemoveContainer" containerID="b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0" Oct 30 22:08:45.064432 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.064413 6808 scope.go:122] "RemoveContainer" containerID="b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0" Oct 30 22:08:45.064683 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:08:45.064664 6808 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0\": container with ID starting with b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0 not found: ID does not exist" containerID="b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0" Oct 30 22:08:45.064732 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.064691 6808 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0"} err="failed to get container status \"b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0\": rpc error: code = NotFound desc = could not find container \"b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0\": container with ID starting with b72a325264583049a04238387a6a2fb5b6901f5a9d34f568cabb9acc7f233ff0 not found: ID does not exist" Oct 30 22:08:45.077525 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.077503 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq"] Oct 30 22:08:45.089855 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.089832 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a86449bf-kserve-6cd9469d69-vjrhq"] Oct 30 22:08:45.926560 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:08:45.926528 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99191bf6-7c89-4200-ba11-1dcbcee37c3d" path="/var/lib/kubelet/pods/99191bf6-7c89-4200-ba11-1dcbcee37c3d/volumes" Oct 30 22:09:05.701528 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:09:05.701487 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:09:07.441051 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.441010 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx"] Oct 30 22:09:07.441552 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.441530 6808 cpu_manager.go:414] "containerMap: removing container" podUID="99191bf6-7c89-4200-ba11-1dcbcee37c3d" containerName="main" Oct 30 22:09:07.441901 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.441555 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99191bf6-7c89-4200-ba11-1dcbcee37c3d" containerName="main" Oct 30 22:09:07.441901 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.441697 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="99191bf6-7c89-4200-ba11-1dcbcee37c3d" containerName="main" Oct 30 22:09:07.445767 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.445747 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.469773 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.469746 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:09:07.470269 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.470240 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:07.478959 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.470469 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:07.478959 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.470503 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4ttw5\"" Oct 30 22:09:07.478959 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.473047 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:09:07.480373 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.480345 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx"] Oct 30 22:09:07.537994 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.537970 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhh8w\" (UniqueName: \"kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.538135 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.538038 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.538135 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.538120 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.538232 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.538152 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.639062 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.639029 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.639215 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.639077 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.639215 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.639188 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.639341 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.639264 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhh8w\" (UniqueName: \"kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.639684 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.639658 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.641378 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.641348 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.641485 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.641438 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.651385 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.651366 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhh8w\" (UniqueName: \"kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w\") pod \"maas-api-6884d9c49d-gzrsx\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:07.758661 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:07.758595 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:08.037683 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:08.037654 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx"] Oct 30 22:09:08.040422 ip-10-0-137-168 kubenswrapper[6808]: W1030 22:09:08.040384 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f6cb36_ade7_44b9_8258_f3fe48743217.slice/crio-f06d4d59871e56fd871bad5409c03c59bf89c5c605516c9aa3d3d8e817a01677 WatchSource:0}: Error finding container f06d4d59871e56fd871bad5409c03c59bf89c5c605516c9aa3d3d8e817a01677: Status 404 returned error can't find the container with id f06d4d59871e56fd871bad5409c03c59bf89c5c605516c9aa3d3d8e817a01677 Oct 30 22:09:08.203591 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:08.203555 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" event={"ID":"f8f6cb36-ade7-44b9-8258-f3fe48743217","Type":"ContainerStarted","Data":"f06d4d59871e56fd871bad5409c03c59bf89c5c605516c9aa3d3d8e817a01677"} Oct 30 22:09:09.209991 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:09.209940 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" event={"ID":"f8f6cb36-ade7-44b9-8258-f3fe48743217","Type":"ContainerStarted","Data":"156653280a491e84c5eee525097235f8de8444865f08c0cfc71c006c89dc0c46"} Oct 30 22:09:09.210430 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:09.210079 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:09.237687 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:09.237642 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" podStartSLOduration=1.7728638810000001 podStartE2EDuration="2.237630517s" totalImagesPullingTime="464.766636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:09.237155245 +0000 UTC m=+2411.432846368" watchObservedRunningTime="2026-10-30 22:09:09.237630517 +0000 UTC m=+2411.433321635" Oct 30 22:09:15.024987 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.024936 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx"] Oct 30 22:09:15.025593 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.025202 6808 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" podUID="f8f6cb36-ade7-44b9-8258-f3fe48743217" containerName="maas-api" containerID="cri-o://156653280a491e84c5eee525097235f8de8444865f08c0cfc71c006c89dc0c46" gracePeriod=30 Oct 30 22:09:15.031215 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.031193 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:15.074044 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.074013 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d"] Oct 30 22:09:15.077440 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.077418 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.091357 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.091321 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d"] Oct 30 22:09:15.109557 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.109528 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ab3f5-fd32-4ccc-8758-2d7989f23647-maas-api-tls\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.109699 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.109567 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ab3f5-fd32-4ccc-8758-2d7989f23647-openshift-service-ca\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.109699 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.109666 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ab3f5-fd32-4ccc-8758-2d7989f23647-metrics-certs\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.109817 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.109761 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n475\" (UniqueName: \"kubernetes.io/projected/d10ab3f5-fd32-4ccc-8758-2d7989f23647-kube-api-access-6n475\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.211634 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.211598 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n475\" (UniqueName: \"kubernetes.io/projected/d10ab3f5-fd32-4ccc-8758-2d7989f23647-kube-api-access-6n475\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.211810 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.211676 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ab3f5-fd32-4ccc-8758-2d7989f23647-maas-api-tls\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.211810 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.211705 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ab3f5-fd32-4ccc-8758-2d7989f23647-openshift-service-ca\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.211810 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.211772 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ab3f5-fd32-4ccc-8758-2d7989f23647-metrics-certs\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.213682 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.213651 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ab3f5-fd32-4ccc-8758-2d7989f23647-openshift-service-ca\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.218007 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.214984 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ab3f5-fd32-4ccc-8758-2d7989f23647-metrics-certs\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.219704 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.219676 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ab3f5-fd32-4ccc-8758-2d7989f23647-maas-api-tls\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.238675 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.238639 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n475\" (UniqueName: \"kubernetes.io/projected/d10ab3f5-fd32-4ccc-8758-2d7989f23647-kube-api-access-6n475\") pod \"maas-api-6fdb49dd46-nds6d\" (UID: \"d10ab3f5-fd32-4ccc-8758-2d7989f23647\") " pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.246044 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.244993 6808 generic.go:356] "Generic (PLEG): container finished" podID="f8f6cb36-ade7-44b9-8258-f3fe48743217" containerID="156653280a491e84c5eee525097235f8de8444865f08c0cfc71c006c89dc0c46" exitCode=0 Oct 30 22:09:15.246044 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.245024 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" event={"ID":"f8f6cb36-ade7-44b9-8258-f3fe48743217","Type":"ContainerDied","Data":"156653280a491e84c5eee525097235f8de8444865f08c0cfc71c006c89dc0c46"} Oct 30 22:09:15.286883 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.286812 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:15.312257 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.312227 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs\") pod \"f8f6cb36-ade7-44b9-8258-f3fe48743217\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " Oct 30 22:09:15.312410 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.312291 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w\" (UniqueName: \"kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w\") pod \"f8f6cb36-ade7-44b9-8258-f3fe48743217\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " Oct 30 22:09:15.312410 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.312374 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca\") pod \"f8f6cb36-ade7-44b9-8258-f3fe48743217\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " Oct 30 22:09:15.312410 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.312400 6808 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls\") pod \"f8f6cb36-ade7-44b9-8258-f3fe48743217\" (UID: \"f8f6cb36-ade7-44b9-8258-f3fe48743217\") " Oct 30 22:09:15.312764 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.312736 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca" pod "f8f6cb36-ade7-44b9-8258-f3fe48743217" (UID: "f8f6cb36-ade7-44b9-8258-f3fe48743217"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:15.314448 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.314423 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs" pod "f8f6cb36-ade7-44b9-8258-f3fe48743217" (UID: "f8f6cb36-ade7-44b9-8258-f3fe48743217"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:15.314561 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.314530 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w" pod "f8f6cb36-ade7-44b9-8258-f3fe48743217" (UID: "f8f6cb36-ade7-44b9-8258-f3fe48743217"). InnerVolumeSpecName "kube-api-access-vhh8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:15.314625 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.314568 6808 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls" pod "f8f6cb36-ade7-44b9-8258-f3fe48743217" (UID: "f8f6cb36-ade7-44b9-8258-f3fe48743217"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:15.413860 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.413819 6808 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f6cb36-ade7-44b9-8258-f3fe48743217-openshift-service-ca\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:15.413860 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.413847 6808 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-maas-api-tls\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:15.413860 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.413857 6808 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f6cb36-ade7-44b9-8258-f3fe48743217-metrics-certs\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:15.413860 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.413866 6808 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhh8w\" (UniqueName: \"kubernetes.io/projected/f8f6cb36-ade7-44b9-8258-f3fe48743217-kube-api-access-vhh8w\") on node \"ip-10-0-137-168.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:15.417650 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.417621 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:15.765626 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:15.765597 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d"] Oct 30 22:09:15.770903 ip-10-0-137-168 kubenswrapper[6808]: W1030 22:09:15.770874 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10ab3f5_fd32_4ccc_8758_2d7989f23647.slice/crio-6864dd391e1ff92f2c27391d89accef0f500c8af62facd105faadf2fda7bb7ea WatchSource:0}: Error finding container 6864dd391e1ff92f2c27391d89accef0f500c8af62facd105faadf2fda7bb7ea: Status 404 returned error can't find the container with id 6864dd391e1ff92f2c27391d89accef0f500c8af62facd105faadf2fda7bb7ea Oct 30 22:09:16.250801 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:16.250763 6808 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" Oct 30 22:09:16.250801 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:16.250778 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx" event={"ID":"f8f6cb36-ade7-44b9-8258-f3fe48743217","Type":"ContainerDied","Data":"f06d4d59871e56fd871bad5409c03c59bf89c5c605516c9aa3d3d8e817a01677"} Oct 30 22:09:16.251305 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:16.250839 6808 scope.go:122] "RemoveContainer" containerID="156653280a491e84c5eee525097235f8de8444865f08c0cfc71c006c89dc0c46" Oct 30 22:09:16.252215 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:16.252194 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" event={"ID":"d10ab3f5-fd32-4ccc-8758-2d7989f23647","Type":"ContainerStarted","Data":"6864dd391e1ff92f2c27391d89accef0f500c8af62facd105faadf2fda7bb7ea"} Oct 30 22:09:16.278933 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:16.278906 6808 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx"] Oct 30 22:09:16.282973 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:16.282920 6808 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6884d9c49d-gzrsx"] Oct 30 22:09:17.258917 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:17.258877 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" event={"ID":"d10ab3f5-fd32-4ccc-8758-2d7989f23647","Type":"ContainerStarted","Data":"9d68de756a42a9a10a92cea8a433da2b5acd3e5b6dc6d241d8a810cca80b9d67"} Oct 30 22:09:17.259338 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:17.258976 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:17.291037 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:17.290995 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" podStartSLOduration=1.833570023 podStartE2EDuration="2.290982664s" totalImagesPullingTime="457.412641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:17.290310988 +0000 UTC m=+2419.486002110" watchObservedRunningTime="2026-10-30 22:09:17.290982664 +0000 UTC m=+2419.486673783" Oct 30 22:09:17.933364 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:17.933333 6808 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f6cb36-ade7-44b9-8258-f3fe48743217" path="/var/lib/kubelet/pods/f8f6cb36-ade7-44b9-8258-f3fe48743217/volumes" Oct 30 22:09:23.269614 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:09:23.269582 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fdb49dd46-nds6d" Oct 30 22:09:35.698697 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:09:35.698653 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:10:05.698280 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:10:05.698238 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:10:35.697905 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:10:35.697870 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:11:05.697510 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:11:05.697476 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:11:35.697831 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:11:35.697784 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:12:05.697865 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:12:05.697827 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:12:35.696894 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:12:35.696858 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:12:43.401487 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:12:43.401391 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-794b4996f-5679g_c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d/authorino/0.log" Oct 30 22:12:54.912561 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:12:54.912530 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ckd5_e595af5c-5533-44ac-b2ac-87f30ea41af7/manager/0.log" Oct 30 22:12:55.160583 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:12:55.160558 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bff7d9b4d-4ftf4_10aa493a-209e-4052-bed7-47319c308928/manager/0.log" Oct 30 22:12:55.763184 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:12:55.763151 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcv85_d4c20eec-664a-4952-9368-7f8e414ce138/proxy/0.log" Oct 30 22:12:56.519597 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:12:56.519560 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fdb49dd46-nds6d_d10ab3f5-fd32-4ccc-8758-2d7989f23647/maas-api/0.log" Oct 30 22:12:59.587820 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:12:59.587786 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-794b4996f-5679g_c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d/authorino/0.log" Oct 30 22:13:01.816612 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:01.816585 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rc7lv_e971dbae-c1c7-499f-9b1d-79789f4c3769/discovery/0.log" Oct 30 22:13:02.007098 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:02.007068 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-gjrcd_81dc933c-7be6-4b97-836d-511e02294976/kube-auth-proxy/0.log" Oct 30 22:13:03.530149 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:03.530122 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hxfjm_ce40587d-41ac-4215-9c6c-f0298ebec6a6/istio-proxy/0.log" Oct 30 22:13:03.909532 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:03.909508 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w8ptl_55d693b2-3c4e-48e2-977f-d4cbef44f3b3/istio-proxy/0.log" Oct 30 22:13:04.118279 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:04.118252 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-wtthn_c3699ff6-469e-42ec-9f4d-c9e124f1f3cc/istio-proxy/0.log" Oct 30 22:13:05.697443 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:13:05.697406 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:13:07.854528 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:07.854497 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-794b4996f-5679g_c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d/authorino/0.log" Oct 30 22:13:17.790655 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:17.790623 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ckd5_e595af5c-5533-44ac-b2ac-87f30ea41af7/manager/0.log" Oct 30 22:13:18.027765 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:18.027737 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bff7d9b4d-4ftf4_10aa493a-209e-4052-bed7-47319c308928/manager/0.log" Oct 30 22:13:18.660452 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:18.660425 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcv85_d4c20eec-664a-4952-9368-7f8e414ce138/proxy/0.log" Oct 30 22:13:19.232125 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:19.232101 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fdb49dd46-nds6d_d10ab3f5-fd32-4ccc-8758-2d7989f23647/maas-api/0.log" Oct 30 22:13:21.927226 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:21.927196 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-794b4996f-5679g_c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d/authorino/0.log" Oct 30 22:13:23.908227 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:23.908197 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rc7lv_e971dbae-c1c7-499f-9b1d-79789f4c3769/discovery/0.log" Oct 30 22:13:24.096787 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:24.096760 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-gjrcd_81dc933c-7be6-4b97-836d-511e02294976/kube-auth-proxy/0.log" Oct 30 22:13:25.574645 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:25.574616 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hxfjm_ce40587d-41ac-4215-9c6c-f0298ebec6a6/istio-proxy/0.log" Oct 30 22:13:25.964029 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:25.963999 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w8ptl_55d693b2-3c4e-48e2-977f-d4cbef44f3b3/istio-proxy/0.log" Oct 30 22:13:26.153908 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:13:26.153881 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-wtthn_c3699ff6-469e-42ec-9f4d-c9e124f1f3cc/istio-proxy/0.log" Oct 30 22:13:35.697068 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:13:35.697032 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:14:05.698113 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:14:05.698073 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:14:07.546408 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:14:07.546376 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bff7d9b4d-4ftf4_10aa493a-209e-4052-bed7-47319c308928/manager/0.log" Oct 30 22:14:07.815988 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:14:07.815891 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bff7d9b4d-4ftf4_10aa493a-209e-4052-bed7-47319c308928/manager/0.log" Oct 30 22:14:35.697218 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:14:35.697181 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:14:57.508066 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:14:57.508035 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bff7d9b4d-4ftf4_10aa493a-209e-4052-bed7-47319c308928/manager/0.log" Oct 30 22:14:57.763323 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:14:57.763240 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bff7d9b4d-4ftf4_10aa493a-209e-4052-bed7-47319c308928/manager/0.log" Oct 30 22:15:02.642178 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:02.642145 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mq6jf_5f8be18f-c1ce-46ae-b36c-df83e438c101/global-pull-secret-syncer/0.log" Oct 30 22:15:02.884399 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:02.884367 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mcglb_9e6b899c-d914-4adb-9a7c-230e359b1497/konnectivity-agent/0.log" Oct 30 22:15:02.933692 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:02.933627 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-168.us-east-2.compute.internal_de6e1dbd7017e381fdeb008c14987499/haproxy/0.log" Oct 30 22:15:05.697871 ip-10-0-137-168 kubenswrapper[6808]: E1030 22:15:05.697836 6808 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8lm4w" containerName="extractor" Oct 30 22:15:09.509311 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:09.509279 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-794b4996f-5679g_c630ee4a-57f1-4e4d-b293-0bd0ad06cb0d/authorino/0.log" Oct 30 22:15:11.101187 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.101154 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/alertmanager/0.log" Oct 30 22:15:11.144907 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.144882 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/config-reloader/0.log" Oct 30 22:15:11.179202 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.179181 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/kube-rbac-proxy-web/0.log" Oct 30 22:15:11.207758 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.207734 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/kube-rbac-proxy/0.log" Oct 30 22:15:11.232927 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.232898 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/kube-rbac-proxy-metric/0.log" Oct 30 22:15:11.258117 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.258098 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/prom-label-proxy/0.log" Oct 30 22:15:11.285426 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.285406 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58f25f80-76a5-4afb-8b92-00ca20f9ebc9/init-config-reloader/0.log" Oct 30 22:15:11.371458 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.371406 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qf7hf_136a5087-3017-46af-aa81-15b1a899031a/kube-state-metrics/0.log" Oct 30 22:15:11.393107 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.393084 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qf7hf_136a5087-3017-46af-aa81-15b1a899031a/kube-rbac-proxy-main/0.log" Oct 30 22:15:11.418848 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.418824 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qf7hf_136a5087-3017-46af-aa81-15b1a899031a/kube-rbac-proxy-self/0.log" Oct 30 22:15:11.461343 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.461324 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bdd5667d8-qdt5r_0328b8ce-272d-4232-8681-ef9e0d3a5274/metrics-server/0.log" Oct 30 22:15:11.490612 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.490593 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-zgn2r_129d99fe-11f5-478f-b470-0908f5a42037/monitoring-plugin/0.log" Oct 30 22:15:11.530763 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.530745 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8psws_40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/node-exporter/0.log" Oct 30 22:15:11.561632 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.561610 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8psws_40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/kube-rbac-proxy/0.log" Oct 30 22:15:11.587204 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.587184 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8psws_40ecfbd7-6c2e-42ee-a7dd-73a91475f5c9/init-textfile/0.log" Oct 30 22:15:11.742531 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.742506 6808 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw"] Oct 30 22:15:11.742920 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.742905 6808 cpu_manager.go:414] "containerMap: removing container" podUID="f8f6cb36-ade7-44b9-8258-f3fe48743217" containerName="maas-api" Oct 30 22:15:11.742997 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.742922 6808 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f6cb36-ade7-44b9-8258-f3fe48743217" containerName="maas-api" Oct 30 22:15:11.743086 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.743039 6808 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f6cb36-ade7-44b9-8258-f3fe48743217" containerName="maas-api" Oct 30 22:15:11.746498 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.746481 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.758547 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.758525 6808 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vrzqz\"/\"default-dockercfg-p8sks\"" Oct 30 22:15:11.760701 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.760683 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw"] Oct 30 22:15:11.772989 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.772942 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-lib-modules\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.773113 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.773090 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-podres\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.773157 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.773142 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-sys\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.773193 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.773170 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h985s\" (UniqueName: \"kubernetes.io/projected/ce32cf6f-c64a-4aff-b243-e352ee2526a6-kube-api-access-h985s\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.773234 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.773222 6808 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-proc\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.783465 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.783440 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vrzqz\"/\"kube-root-ca.crt\"" Oct 30 22:15:11.783703 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.783687 6808 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vrzqz\"/\"openshift-service-ca.crt\"" Oct 30 22:15:11.821171 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.821148 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-csz4z_cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/kube-rbac-proxy-main/0.log" Oct 30 22:15:11.848106 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.848073 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-csz4z_cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/kube-rbac-proxy-self/0.log" Oct 30 22:15:11.872648 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.872625 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-csz4z_cfdaf21b-e4a6-4c90-8eb2-4dbf75dfa59f/openshift-state-metrics/0.log" Oct 30 22:15:11.873757 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873739 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-podres\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.873844 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873775 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-sys\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.873844 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873798 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h985s\" (UniqueName: \"kubernetes.io/projected/ce32cf6f-c64a-4aff-b243-e352ee2526a6-kube-api-access-h985s\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.873844 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873832 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-proc\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.873963 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873860 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-podres\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.873963 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873864 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-sys\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.873963 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873916 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-proc\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.874072 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.873944 6808 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-lib-modules\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.874072 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.874037 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce32cf6f-c64a-4aff-b243-e352ee2526a6-lib-modules\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:11.886889 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:11.886860 6808 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h985s\" (UniqueName: \"kubernetes.io/projected/ce32cf6f-c64a-4aff-b243-e352ee2526a6-kube-api-access-h985s\") pod \"perf-node-gather-daemonset-zzhxw\" (UID: \"ce32cf6f-c64a-4aff-b243-e352ee2526a6\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:12.059009 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.058929 6808 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:12.318005 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.317922 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f995ff688-xs5n7_b4e22dd8-f90b-4512-9ade-3f3ea3329946/thanos-query/0.log" Oct 30 22:15:12.334660 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.334632 6808 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw"] Oct 30 22:15:12.338448 ip-10-0-137-168 kubenswrapper[6808]: W1030 22:15:12.338420 6808 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce32cf6f_c64a_4aff_b243_e352ee2526a6.slice/crio-1f5340c6b223db7cd8c23a94f9a902a14c3bfda835b06cd84adf765716b5de0b WatchSource:0}: Error finding container 1f5340c6b223db7cd8c23a94f9a902a14c3bfda835b06cd84adf765716b5de0b: Status 404 returned error can't find the container with id 1f5340c6b223db7cd8c23a94f9a902a14c3bfda835b06cd84adf765716b5de0b Oct 30 22:15:12.353672 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.353654 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f995ff688-xs5n7_b4e22dd8-f90b-4512-9ade-3f3ea3329946/kube-rbac-proxy-web/0.log" Oct 30 22:15:12.381027 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.381009 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f995ff688-xs5n7_b4e22dd8-f90b-4512-9ade-3f3ea3329946/kube-rbac-proxy/0.log" Oct 30 22:15:12.408178 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.408157 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f995ff688-xs5n7_b4e22dd8-f90b-4512-9ade-3f3ea3329946/prom-label-proxy/0.log" Oct 30 22:15:12.436294 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.436274 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f995ff688-xs5n7_b4e22dd8-f90b-4512-9ade-3f3ea3329946/kube-rbac-proxy-rules/0.log" Oct 30 22:15:12.463476 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.463460 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f995ff688-xs5n7_b4e22dd8-f90b-4512-9ade-3f3ea3329946/kube-rbac-proxy-metrics/0.log" Oct 30 22:15:12.863645 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.863611 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" event={"ID":"ce32cf6f-c64a-4aff-b243-e352ee2526a6","Type":"ContainerStarted","Data":"5d2aa780ef9fc720984987f80328f2398851100bca660e96cf90d54741e267a9"} Oct 30 22:15:12.863855 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.863671 6808 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" event={"ID":"ce32cf6f-c64a-4aff-b243-e352ee2526a6","Type":"ContainerStarted","Data":"1f5340c6b223db7cd8c23a94f9a902a14c3bfda835b06cd84adf765716b5de0b"} Oct 30 22:15:12.863855 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.863703 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:12.883482 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:12.883430 6808 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" podStartSLOduration=1.8834165889999999 podStartE2EDuration="1.883416589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:12.883085338 +0000 UTC m=+2775.078776458" watchObservedRunningTime="2026-10-30 22:15:12.883416589 +0000 UTC m=+2775.079107705" Oct 30 22:15:14.780879 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:14.780851 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55bf7567fb-64mnn_bdd187cf-c3a9-4109-8c0a-a003faccb300/console/0.log" Oct 30 22:15:16.752341 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:16.752314 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xfdq8_8d852c03-e50d-4039-bd52-7bfae081a271/dns/0.log" Oct 30 22:15:16.781505 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:16.781486 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xfdq8_8d852c03-e50d-4039-bd52-7bfae081a271/kube-rbac-proxy/0.log" Oct 30 22:15:16.872502 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:16.872477 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7866_9233cc4d-86f9-4890-9464-33c7eefd42a8/dns-node-resolver/0.log" Oct 30 22:15:17.587106 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:17.587080 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8l22v_40ae92ae-3d8d-4ea7-bb9f-88d481fc0ada/node-ca/0.log" Oct 30 22:15:18.878546 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:18.878515 6808 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-zzhxw" Oct 30 22:15:19.779032 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:19.778999 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rc7lv_e971dbae-c1c7-499f-9b1d-79789f4c3769/discovery/0.log" Oct 30 22:15:20.436563 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:20.436536 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-gjrcd_81dc933c-7be6-4b97-836d-511e02294976/kube-auth-proxy/0.log" Oct 30 22:15:20.732691 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:20.732614 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hxfjm_ce40587d-41ac-4215-9c6c-f0298ebec6a6/istio-proxy/0.log" Oct 30 22:15:20.804693 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:20.804669 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w8ptl_55d693b2-3c4e-48e2-977f-d4cbef44f3b3/istio-proxy/0.log" Oct 30 22:15:20.839631 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:20.839605 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-wtthn_c3699ff6-469e-42ec-9f4d-c9e124f1f3cc/istio-proxy/0.log" Oct 30 22:15:22.966727 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:22.966702 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8btfx_a9f17f5b-18dc-4642-a2bb-d778b0524561/serve-healthcheck-canary/0.log" Oct 30 22:15:23.341119 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.341028 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rc7lv_e971dbae-c1c7-499f-9b1d-79789f4c3769/discovery/0.log" Oct 30 22:15:23.353266 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.353244 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-gjrcd_81dc933c-7be6-4b97-836d-511e02294976/kube-auth-proxy/0.log" Oct 30 22:15:23.555772 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.555745 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hxfjm_ce40587d-41ac-4215-9c6c-f0298ebec6a6/istio-proxy/0.log" Oct 30 22:15:23.596466 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.596410 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w8ptl_55d693b2-3c4e-48e2-977f-d4cbef44f3b3/istio-proxy/0.log" Oct 30 22:15:23.615466 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.615446 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-wtthn_c3699ff6-469e-42ec-9f4d-c9e124f1f3cc/istio-proxy/0.log" Oct 30 22:15:23.680022 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.680005 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8lm4w_c4693396-79bd-410d-86a3-5a61f614d231/kube-rbac-proxy/0.log" Oct 30 22:15:23.706155 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.706137 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8lm4w_c4693396-79bd-410d-86a3-5a61f614d231/exporter/0.log" Oct 30 22:15:23.736560 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:23.736535 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8lm4w_c4693396-79bd-410d-86a3-5a61f614d231/extractor/0.log" Oct 30 22:15:26.362060 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:26.362032 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bbc596d7f-bxrkp_be5ce777-dd85-42cd-a303-58b060dd092f/manager/0.log" Oct 30 22:15:26.389245 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:26.389220 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bbc596d7f-bxrkp_be5ce777-dd85-42cd-a303-58b060dd092f/copy-manifests/0.log" Oct 30 22:15:27.106254 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:27.106225 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rc7lv_e971dbae-c1c7-499f-9b1d-79789f4c3769/discovery/0.log" Oct 30 22:15:27.142058 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:27.142031 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-gjrcd_81dc933c-7be6-4b97-836d-511e02294976/kube-auth-proxy/0.log" Oct 30 22:15:27.304748 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:27.304720 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hxfjm_ce40587d-41ac-4215-9c6c-f0298ebec6a6/istio-proxy/0.log" Oct 30 22:15:27.345604 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:27.345583 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w8ptl_55d693b2-3c4e-48e2-977f-d4cbef44f3b3/istio-proxy/0.log" Oct 30 22:15:27.366699 ip-10-0-137-168 kubenswrapper[6808]: I1030 22:15:27.366639 6808 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-wtthn_c3699ff6-469e-42ec-9f4d-c9e124f1f3cc/istio-proxy/0.log"